Skip to Content

Improving Pantheon’s First Time User Experience

In this project to improve Pantheon’s first time user experience (FTUE), we began with user research. Our primary user group are agencies developers building and maintaining sites for clients. We interviewed newly registered users to learn:

  1. Motivations for trying Pantheon
  2. Their product experience, especially where they encounter roadblocks.

During interviews, we learned users are motivated to try Pantheon to solve these common problems:

  • Setting up the same infrastructure for every new project wastes time and non-billable hours. They need a solution to work out-of-the-box so that they can start building the site their client is paying them to build.
    • For example, users know developing a on live site is bad practice and can cause breakage, but there is high investment upfront to develop better by setting up things like staging environments. They often don’t have time to make this investment, or don’t know how, so they continue to work this way.
  • Core updates for WordPress and Drupal can be a pain to apply to a site. Rolling back updates that have gone wrong is even more of a pain. Users want to get unexciting and potentially stressful work like this off their plates.
  • Users need a solution that is reliable. One user told us he doesn’t want his client calling him in the middle of the night because the site doesn’t work, so that he can get a full night’s sleep. And practices like developing on a live site makes it more likely to cause this client-induced sleep deprivation!

In discussing their product experience, we identified a few common roadblocks encountered within the first few minutes:

  • After creating their first site, users land on this dashboard. Users often know that the next step they need to do is install their chosen CMS and access the code repository. It’s unclear how to do either of these.

None of the existing copy or design leads users to the next steps they should take.

  • Pantheon’s isolated environments for Development and Testing are ready to be used out-of-the-box which is highly valuable, but users are unclear when and how to use each environment.

  • When users land on the site dashboard for the first time, it is likely they will be overwhelmed with all of the options to choose from – there are 30 interaction choices and no guidance. Users who have no particular task in mind are even more likely to be paralyzed.

 


Prototypes

With these learnings in mind, we prototyped several approaches to improve the first time experience for users and increase user retention in their first week. Below is a sample.

Prototype 1: Site development flow

We know that users who make a first code commit are more likely to stick with Pantheon as active users, and we know from our analytics that of all users who make at least 1 code commit, 56% do so in the first 4 days of registration. There is a sharp fall off after the first 4 days, so a user’s first few days using the product is a critical time to demonstrate how they can work better.

Having identified the major, immediate roadblocks to starting site development – how to install the CMS, and how to access the site code – this prototype redesigns the flow to reduce these roadblocks.

Hypothesis: If we make the path leading to site development more straightforward, through guidance and reinforcing success, it will be more likely new users will become active users and therefore return in Week 0.

Remember when users land on the dashboard after creating the site, there is no direction on what to do next.  The prototype picks up after site spin-up to improve this:

After creating their first site, the prototype directed users to install the CMS (here, WordPress) right away.

Once WordPress was installed, we gave them a success confirmation, which we know from past observation users were looking for as reinforcement that they had installed the CMS correctly. When a confirmation is not present, users became hesitant to continue. This prototype also includes guidance on a next step.

In the current UI: the command the user needs to clone their repository is buried around a lot of introductory information that most don’t need, making it difficult to locate.

The prototype reduced the dropdown to the bare necessities to try to help users find what they need immediately.

Once users cloned the code repo, made a change, and made the first code commit, the prototype provides additional guidance.

In user testing this prototype, we observed large improvements in users understanding what to do by giving feedback when actions were completed successfully, suggesting next step(s), and then making it obvious how to take those next steps.

While this improved some roadblocks, it introduced other confusion due to some major interface changes. While we determined we would need more time on iteration and testing than we have to move forward with an overhaul like this, some parts of this new flow did make it into the product that you’ll see below.

Prototype 2 – “What do you want to do?”

We learned from the research that users will try Pantheon to improve specific workflow problems. This prototype more proactively communicates how Pantheon can help with those problems.

Hypothesis: By asking what new users came to Pantheon to do, and giving them the ability to select an activity that will move them towards that goal, then it is more likely new users will accomplish their goal and see product benefits, and therefore return in Week 0.

In the current experience after creating an account, users are landed on their home screen. Many users do find there way to an action from here but there’s no real guidance and no highlighting of valuable actions they may want to try first, which is a missed opportunity to hook users right away.

This prototype introduces a new screen shown to new users immediately after registration. We suggest actions that we learned are most common that users do. Using users’ own reasons, we also highlight the benefits of using Pantheon in the description text. If users aren’t sure what to do, we use social proof at the bottom to inform them what most developers do first, to help them decide.

During user testing, this prototype resonated with users because it is upfront about how the product can  address workflow challenges, coupled with an action they can take right now to move them towards that desired future state.

Summary of all prototype testing:

  • Moments that communicate how Pantheon solves problems resonated for new users.
  • Connecting actions together in user flows helped new users start an activity and follow it through.
  • When new users weren’t told what to do next, they got stuck.

Product Changes

From the prototype and research learnings, we synthesized our approach to implementing product improvements:

To increase return rate in Week 0, we should communicate to new users how Pantheon can solve current workflow challenges, and follow through by prompting action so users can make progress towards the problem(s) they want solved.

Tactically, this means:

  1. After registration, immediately communicate with new users how Pantheon will help them and prompt an action to take.
  2. After new users create their first site, provide guidance to the next actions  – install the CMS then clone or connect to the site code – so that they can more quickly start site development and try out our developer tools.
  3. Connect site development activities into a sequence of actions, taking into account observed user behavior, in order to reinforce success and make next steps obvious.

Welcome Screen

The 2nd prototype evolved into the Welcome screen highlighting benefits of using Pantheon to improve workflows.

If users decide to start a new project, they will need to choose their site CMS. We updated this text to be tailored to why it’s better to use each CMS on Pantheon. For example, WordPress users are more likely to not have experience with version control even though they know it’s a good practice – Pantheon’s built-in version control will enable them to easily adopt the practice.

Dev environment empty states

To teach users when and how to use each development environment, we updated the starting empty states to be more concise and focus on the benefit to the user instead of just what the feature is.

This is the current start state of the Dev environment which isn’t a real empty state and doesn’t help the user understand what to do.

For the Dev environment we implemented a real empty state, along with guidance on how to start developing by first installing WordPress

The Test and Live environments follow the same design so that they all look like parts of the same whole. They also got a copy update to better convey when and how to use each environment.

Clone or connect to site code

After installing the CMS, we display the success confirmation.

And now the user likely wants to clone the repo with git or connect by SFTP.

The current UI makes this hard to find.

For this change, we ran an A/B test comparing the current UI and this new design:

In this new design, we tried guiding the user through the UI design by using our primary CTA color on the next button they should use and de-cluttering the area around it. We’ve tried previous A/B tests of guiding users explicitly to the site code through tooltip overlays as part of a tutorial, which didn’t work. This time we tried implicitly guiding them through the UI design to see if this would result in more users finding their way to the site code and successfully making a first code commit.

We also took the simplified dropdown from the prototype test to make it more obvious what to do.

Results for this dropdown test show a 10% improvement for the new design in the number of users who successfully make their first code commit.

As always, there are some ideas that didn’t make it into the implementation.  To continue improving the FTUE in the future, we need to re-test the product with the new changes as well as discover what roadblocks exist in subsequent tasks so that we can continue to help new users progress further in their journey.