Custom Upstreams – Kim Kiser Ramirez

PANTHEON | Custom Upstreams

I led the design of this new feature on Pantheon. Custom upstreams help teams work smarter by reducing site setup and maintenance work by using a common starting codebase for their portfolio of sites.

We started with a grand vision of this feature that would have caused us to bite off more than we could chew. Through many rounds of prototyping, small releases, and iteration based on beta tester feedback, we were able to develop an MVP that met our user and business needs without over-investing unnecessarily. Through testing, we also identified unmet user needs that weren't originally planned for this project. The MVP freed up time to invest in those unmet needs, and we ultimately launched a high quality feature that helped our developers' workflows even more than we had thought.

I created this diagram as part of this project to help new users learn what the feature is and how it can be used.

What is a custom upstream?

When creating a new site on Pantheon, developers can start with one of our WordPress or Drupal installations, or they can deploy a custom installation. Custom installations can contain starter code shared between website projects, which reduces set-up work. This installation acts as an upstream to sites - all updates to the upstream respository can be applied to any site, which also makes maintainence more efficient.

An update to the respository is fed into the upstream on Pantheon.

Custom upstreams are hosted in a repository, using a platform such as GitHub. An update to the repo will feed into the upstream of the site on Pantheon.

The update is ready to apply to my site on my Pantheon site dashboard.

When the commit is made on the repo, the update is staged on the Pantheon site dashboard, ready for the developer to apply the update to the site.

BEFORE

Before this project, using this feature was a high touch process with a high barrier to entry for customers.

Before, developers only received instructions through this UI to contact us to use the feature.

Custom upstreams technically existed as a feature on Pantheon for a while as a high touch capability - our support team had to set up and manage the upstream, which is a high barrier to entry for users and adds support costs for us.

Developers had to send us a large amount of information in a support ticket in order for our support team to set up their upstream.

PROJECT GOAL

We have observed more engagement on Pantheon for teams who used a custom upstream compared with those who don't. We wanted to unlock the potential of other companies and make this self-service, by removing the barrier to entry for customers and reducing our support costs.

PROTOTYPING

We started with user flows and interactive prototypes to describe the initial product vision

The team and I built user flows together to visualize how this should function and identify where to account for errors. View larger

Initially envisioned as a full integration with GitHub and Bitbucket - the two most frequently used platforms for upstream repositories for Pantheon developers - we planned for the developer to authenticate with their GitHub or Bitbucket account and set up their repository within a few clicks.

We prototyped a new "Upstreams" area for developers to manage an existing upstream and create a new site directly from here.

View Interactive Prototype →

We user tested the prototypes to understand which parts of the experience we could confidently move forward with. We documented observed problems and mapped them to our template to determine which usability problems were highest priority to solve.

1st priority: These problems prevent the tester from completing the task, and there is high risk to the project's success if we don't solve these
2nd priority: These problems cause some difficulty, but the tester can ultimately complete the task. These can be addressed after all 1st priority problems have been solved.
3rd priority: These problems don't interfere directly with the tester completing the task, so there is little to no risk to the project's success if we don't solve these.

Thanks to Erika Hall's Just Enough Research for this approach.

Our template for user test analysis.

MVP DEFINITION

As we planned the full experience, I worked with the PO and engineering team to define an MVP we could release so that we can unlock this capability for developers as soon as possible.

First Release

While planning the full GitHub/Bitbucket integration, we planned an MVP release enabling a developer to add a custom upstream manually by entering details into a form (instead of authenticating their account and selecting from their list of repositories). We released version #1 to a group of 15 beta testers.

Once the upstream is created, it is added to the organization's list of upstreams the developers can then manage and start new sites from.

USER TESTING & ITERATION

We user tested with beta testers to observe them using this new flow with real upstreams and sites.

Primary blocker

Most testers found authentication instructions confusing, especially if it is irrelevant to them. Pantheon only needs a token if the upstream repo is set to "private"; if it's a public repo then no password is needed. However users often don't know what the repo privacy setting is, or where to find it. We observed high error rates during this section, and observed several testers staring at this field for several seconds trying to figure out which set of directions they should follow.

Our immediate goal became to reduce the error rate on this form, so that we can be more confident in releasing this to a wider number of developers.

Revised form

In the next release, we simplified the "Add Upstream" form by removing fields the developer was unlikely to change at this time (instead, they are accessible later from the upstream's settings page).

We also simplified the authentication field by hiding it until the system determines whether authentication is needed.

With contextual instructions

The system checks whether the repo is private, and if so, then checks whether to present GitHub or Bitbucket authentications instructions.

We saw great improvement in the time it took to complete this form and a reduction in error rates, due to only presenting only the relevant information to the developer, and only when it's needed.

Due to the success of the form iteration, we opted to keep the form instead of evolving this into the full prototyped experience, because the cost to implement ended up being too high for the benefit it would give developers, because...

1 / The engineering work was much larger scope than originally anticipated. It would have taken multiple sprints to implement and would not save the users much time or work.

2 / Adding an upstream is an infrequent user task; most teams only have 1 or 2 upstreams. Once added to Pantheon, upstreams are maintained from there and do not need to be re-added. In user testing, we observed users completing the new form in only a few seconds.

The team recommended not investing further in the upstream adding experience. Due to this, we were actually able to invest in a few unmet user needs we unearthed during the user testing. These investments improved the quality of this feature for upstreams already in use for live sites, thereby increasing the likelihood that developers who try it will stick with it.

ADDITIONAL USER PROBLEMS

Several unmet user needs were surfaced during the user testing we could now turn our attention to!

1 / Deploying an upstream update to a site is slow and unreliable.

2 / Inability to switch an upstream of a site to another upstream.

Problem #1: Deploying an upstream update to a site is slow and unreliable

During every user test engagement, this was the #1 complaint and source of frustration surfaced by developers.

At the time, it could take anywhere between 1 - 40 minutes between a code update made on GitHub or Bitbucket to show up in their Pantheon dashboards, so that it can be applied to their sites. This is due to a combination of GitHub/Bitbucket/Pantheon code caching and hit rate limits.

This means teams are unable to push updates out in time for client meetings, or push an urgent update in a timely manner (such as a security patch or hotfix).

Pushing updates is a frequent user task because most teams actively work on their upstream, so the few minutes of waiting at each occurance added up fast. We could save developers a significant amount of waiting time by finding a way to alleviate this problem.

BEFORE

On a site dashboard, if an upstream update is available, developers will be notified here and can apply the update to their site.

When a developer pushes an update to their upstream repository and navigate to their Pantheon site dashboard, their expectation is the update is immediately available to apply so that there is no interruption to their workflow. If the update doesn't show up, the developer refreshes the browser periodically until it does.

AFTER

After some technical discovery, the engineering team determined implementing the ideal instant update would not be feasible.

Instead, we implemented a way for a user to manually "Check Now" for updates. Behind the scenes, this clears the upstream cache and forces a check for an upstream update.

We also implemented a new state letting developers know if the site is up to date in order to reduce uncertainty about the upstream status. Developers can also force a check for an upstream update from this state.

Problem #2: Inability to switch an upstream of a site to another upstream.

This is actually a risky operation that could completely break a customer's live site. So why would developers need to do this? A few use cases came up during testing:

1 / If a team started on Bitbucket and then made a business decision to move to GitHub, or vice versa. Therefore, the location of their repository would need to change.

2 / If a team needs to rearchitect their upstream code, they may need to spin up the new upstream from a brand new repository.

3 / Sites may be on a vanilla CMS that predated the team or tech lead. If the sites could be converted to their own custom upstream used on all new sites, it would simplify their maintanence workflow.

By adding this capability, we can help new users of custom upstreams switch existing sites over to a custom upstream (thus increasing utilization of this feature), and support developers' continued use of this feature by not inconveniencing them due to decisions they make outside of Pantheon.

USER TESTING

We user tested early to make sure to align this capability according to developer expectations

QUESTIONS WE NEEDED TO ANSWER

What would the developer's process be for changing a site's upstream?
How does the developer expect Pantheon to faciliate this?
In what ways does the developer expect the system to guide them?
At what point does the developer have enough confidence to switch a live site upstream?

We knew we needed to fill this need, but how should the capability actually work? We considered two approaches and were unsure which would best support user needs. We thought one of these models, or even a combination of both, would be needed, but we went to our users for clarification.

Would model #1 or #2 best support our user needs?

PROTOTYPE TESTING

We tested both approaches in very early prototypes. We didn't actually expect the product to work exactly as was prototyped - these prototypes served more as conversation starters to begin to understand user expectations.

DEVELOPER'S PROCESS

To manage risk, we learned developers would first extensively test a new upstream on a dedicated test site. Once they gained enough confidence the new upstream worked, developers want to switch the site to the new upstream, as opposed to replacing the original upstream. The reason is so the original repository remains available in case they need to revert.

The desired workflow led the team to recommend Model #2.

CLI TESTING

To continue validating this approach and identify problems we weren't thinking of yet, we wanted to get this quickly into a place where we could user test with real upstreams on actual sites.

We approached this by implementing in Pantheon's command line interface (called Terminus), which is lower cost to get new capabilities working compared to the dashboard GUI. This also gets this ability quickly into the hands of our power users - developers who are command line savvy and the target user for this capability. Notably, this was the first time Pantheon evaluated new features in this way.

Using Terminus to switch the upstream for the site named "Kim's WP Custom Upstream"

As we observed beta testers, we wanted to learn what they might fear about using this, and what gave them enough confidence to move forward on a live site. We wanted to be proactive in addressing their fears and give them confidence through documentation and the command line interaction.

FEARS

Losing custom work from the site codebase
Uncertainty when the upstream would be applied to the site - if it's automatic, it would take away the developer's control
Using the wrong upstream

DEVELOPERS HAVE ENOUGH CONFIDENCE TO USE THIS WHEN...

They understand how the operation works under the hood (eg. how does switching the upstream affect the site's custom code?)
They've finished testing on dedicated test sites

DOCUMENTATION PROVIDES GUIDANCE

I worked with our Documentation team to share our research learnings and develop documentation requirements that addressed users' concerns. The doc includes example use cases to help developers identify when this feature could be helpful for them.

We addressed several common concerns, such as how the update gets applied to the site (it's still in the developer's control!).

TERMINUS ITERATION

We also iterated on the Terminus interaction itself. For example, we prevent a developer from inadvertantly making a change that might completely break their site (such as switching from a WordPress to Drupal based upstream which would create code and database conflicts).

The system stops users from switching the CMS.

RELEASE

After further iteration and testing, this feature was released to all customers.

One beta tester told us that being unable to switch upstreams was really holding his team back from switching everything to GitHub, because it would be hours and hours of work to do it manually. This capability unblocked the team and would enable them to accomplish their goal in just a few hours. Immediately after user testing the experimental Terminus feature, this user felt confident enough to start moving their sites as soon as possible.

In releasing the MVP for creating a custom upstream, and investing in unmet needs we identified through user testing, we ultimately launched a high quality product that helped our developers more than we had initially anticipated. Custom upstreams are now self-service for any customer to try out without having to contact the customer support team.