Solid hypotheses need data not HiPPOs

Part six of our 'lessons in CRO' mini-series (6 of 10)

This mini-series covers 10 lessons we have learned from some of our favourite recent experiments (and the results).  If you're facing similar challenges and want to understand why some things work better than others, or you simply want some new ideas to test, we hope these lessons help guide and inspire you with your next experiments.

Here's lesson six:

‍

Lesson #6: Data trumps opinion. Experiment boldly, regardless of HiPPOs (Highest Paid Person’s Opinion)

Results

• +6% uplift to conversion rate
• +6% uplift in revenue per visitor
• +£1.9m additional revenue/annum

The problem

This company’s newly built eCommerce platform was developed with a focus on UX research and user testing. However, when it was launched visitors reported finding the user experience difficult. And this was affecting conversions and revenue.

Further qualitative research, predominantly post-purchase surveys, were conducted with two key questions:

1. What almost stopped you completing a purchase?
2. What would make it easier for you to complete a purchase?

Several responses suggested that visitors wanted to repeat their previous purchase. Having their previous order available would make this easier and speed up their experience.

These responses, supported with heuristic analysis, identified that customers were mainly returning visitors and were highly habitual in their shopping behaviour.

Little did they know that their previous orders were already available - but being kept in a logged-out state up until the point of payment meant that they couldn’t view their past orders.

What we did

User research showed that to increase conversion rate and revenue, we needed to focus on two separate customer types:

• New Visitors
• Returning Visitors (more specifically, customers)

And to make past purchase reordering easier for returning customers.

However, to identify the different customer types and to enable selection of past purchases, we needed visitors to log in at the earliest opportunity.

Enter the test concept…

Moving the login to an earlier position in the customer journey before browsing the product listing pages.

As you might imagine, this concept was a little controversial and went against most UX best practices but was based on a solid hypothesis. It led to a lot of apprehension among stakeholders and there was a growing unwillingness to test it. Thankfully, everyone placed their trust in the research, and allowed the test to be completed - which led to some surprised faces!

The reasons it worked

From a single test, this was a pretty good uplift, especially considering some of the pushback against the idea.

A sizable chunk of revenue would have been missed if the concept sign-off relied upon stakeholder opinion. Not to mention the future test opportunities that would have been lost as a result.

Nobody can accurately predict the outcome of a test and it’s worth testing (almost) all ideas - especially when it’s based on prior research and it raises some eyebrows. We could argue that tests that aren’t raising eyebrows probably aren’t pushing the potential of what’s achievable through experimentation.

Your next experiment?

Conduct your research; qualitative and quantitative.

Test, test, and test.

Don’t let stakeholder opinions stop you from experimenting. This is easier said than done, but it’s paramount that you push the boundaries.