In this project to improve Pantheon’s first time user experience (FTUE), we began with user research. Our primary user group are agencies developers building and maintaining sites for clients. We interviewed newly registered users to learn:
- Motivations for trying Pantheon
- Their product experience, especially where they encounter roadblocks.
During interviews, we learned users are motivated to try Pantheon to solve these common problems:
- Setting up the same infrastructure for every new project wastes time and non-billable hours. They need a solution to work out-of-the-box so that they can start building the site their client is paying them to build.
- For example, users know developing a on live site is bad practice and can cause breakage, but there is high investment upfront to develop better by setting up things like staging environments. They often don’t have time to make this investment, or don’t know how, so they continue to work this way.
- Core updates for WordPress and Drupal can be a pain to apply to a site. Rolling back updates that have gone wrong is even more of a pain. Users want to get unexciting and potentially stressful work like this off their plates.
- Users need a solution that is reliable. One user told us he doesn’t want his client calling him in the middle of the night because the site doesn’t work, so that he can get a full night’s sleep. And practices like developing on a live site makes it more likely to cause this client-induced sleep deprivation!
In discussing their product experience, we identified a few common roadblocks encountered within the first few minutes:
- After creating their first site, users land on this dashboard. Users often know that the next step they need to do is install their chosen CMS and access the code repository. It’s unclear how to do either of these.
- Pantheon’s isolated environments for Development and Testing are ready to be used out-of-the-box which is highly valuable, but users are unclear when and how to use each environment.
- When users land on the site dashboard for the first time, it is likely they will be overwhelmed with all of the options to choose from – there are 30 interaction choices and no guidance. Users who have no particular task in mind are even more likely to be paralyzed.
With these learnings in mind, we prototyped several approaches to improve the first time experience for users and increase user retention in their first week. Below is a sample.
Prototype 1: Site development flow
We know that users who make a first code commit are more likely to stick with Pantheon as active users, and we know from our analytics that of all users who make at least 1 code commit, 56% do so in the first 4 days of registration. There is a sharp fall off after the first 4 days, so a user’s first few days using the product is a critical time to demonstrate how they can work better.
Having identified the major, immediate roadblocks to starting site development – how to install the CMS, and how to access the site code – this prototype redesigns the flow to reduce these roadblocks.
Hypothesis: If we make the path leading to site development more straightforward, through guidance and reinforcing success, it will be more likely new users will become active users and therefore return in Week 0.
Remember when users land on the dashboard after creating the site, there is no direction on what to do next. The prototype picks up after site spin-up to improve this:
In user testing this prototype, we observed large improvements in users understanding what to do by giving feedback when actions were completed successfully, suggesting next step(s), and then making it obvious how to take those next steps.
While this improved some roadblocks, it introduced other confusion due to some major interface changes. While we determined we would need more time on iteration and testing than we have to move forward with an overhaul like this, some parts of this new flow did make it into the product that you’ll see below.
Prototype 2 – “What do you want to do?”
We learned from the research that users will try Pantheon to improve specific workflow problems. This prototype more proactively communicates how Pantheon can help with those problems.
Hypothesis: By asking what new users came to Pantheon to do, and giving them the ability to select an activity that will move them towards that goal, then it is more likely new users will accomplish their goal and see product benefits, and therefore return in Week 0.
During user testing, this prototype resonated with users because it is upfront about how the product can address workflow challenges, coupled with an action they can take right now to move them towards that desired future state.
Summary of all prototype testing:
- Moments that communicate how Pantheon solves problems resonated for new users.
- Connecting actions together in user flows helped new users start an activity and follow it through.
- When new users weren’t told what to do next, they got stuck.
From the prototype and research learnings, we synthesized our approach to implementing product improvements:
To increase return rate in Week 0, we should communicate to new users how Pantheon can solve current workflow challenges, and follow through by prompting action so users can make progress towards the problem(s) they want solved.
Tactically, this means:
- After registration, immediately communicate with new users how Pantheon will help them and prompt an action to take.
- After new users create their first site, provide guidance to the next actions – install the CMS then clone or connect to the site code – so that they can more quickly start site development and try out our developer tools.
- Connect site development activities into a sequence of actions, taking into account observed user behavior, in order to reinforce success and make next steps obvious.
Dev environment empty states
To teach users when and how to use each development environment, we updated the starting empty states to be more concise and focus on the benefit to the user instead of just what the feature is.
Clone or connect to site code
And now the user likely wants to clone the repo with git or connect by SFTP.
For this change, we ran an A/B test comparing the current UI and this new design:
Results for this dropdown test show a 10% improvement for the new design in the number of users who successfully make their first code commit.
As always, there are some ideas that didn’t make it into the implementation. To continue improving the FTUE in the future, we need to re-test the product with the new changes as well as discover what roadblocks exist in subsequent tasks so that we can continue to help new users progress further in their journey.