Product Growth Experiments Part 1: Concept and Hypothesis

We’re all biased. Often, we tend to look at a given subject based on our experiences or hunches. The solutions we see conform to the way we think, so they must be better. I’ve seen this situation become more and more common, especially in terms of the design or layout.

One of the easiest ways to avoid this situation is to create an A/B test that will confirm or reject your idea. That’s why you often hear, “we can always A/B test that”, as the best solution when there is uncertainty regarding a new design or other disagreements.

Well, you can’t, and you shouldn’t. Or at least you shouldn’t until you do a few things first so you can avoid wasting weeks on experiments that won’t bring you any conclusive results or new information.

The experiment has three distinct stages:

1. Concept and hypothesis (this post)
2. Proper analysis of results
3. Reaching conclusions and documenting insights

Without properly applying all three stages, you won’t be able to benefit from your experiments and you’ll be denied the ensuing growth process. Your team or organization will also miss out on the opportunity and, even worse, your company will lose money, time, and resources because of it.

Below you’ll find the complete guide on how to nail the experimentation process. Enjoy!

---

The single worst thing you can do that will waste your company’s time and money is to start working on the experiment without a clear business plan, measurable goal, and underlying hypothesis. Goals like wanting to increase the number of sales or conversions are common, but it’s not enough to build up an experimentation engine that increases your long-term conversion rates.

After years of experimenting, talking with others, and following various companies that conduct experiments frequently, I created a shortlist with what you need to do before working on the experiment. These items are:

• The big hypothesis
• Experiment concept
• Pre-test calculations
• Quality assurance checklist (optional)

Preparing those items won’t take you more than 20 minutes, but it will force you to really think about the expected business outcome you’re aiming for. Not doing so puts you at risk of wasting your time and resources on something that won’t produce any new information or generate acceptable results.

Let’s jump straight into it.

What is the big hypothesis?

The big hypothesis is a statement that explains why you want to do the experiment, what business outcome it should achieve, and why it should succeed. It should be able to be understood by anyone. Here is a template for creating your hypothesis statement:

We know that (quantitative and qualitative data that backs up why we want to do the experiment. The data should be objective and refer to the business/research perspective).

We believe that (lever/solution/change) for (audience/segment) will result in (goal).

We’ll know by testing (concept) on (area) and observing (KPIs) for (MDEs + minimal sample size).

Preparing the entire experiment concept in such a way requires approaching the upcoming work in a more structured and objective way. At the same time, it is designed for future documentation.

We know that…

The first part forces you to think critically about your idea and:

• Find any data or information that backs up why you should do the experiment at all.
• Analyze the current baselines that you want to improve.
• Go through your previous research and information to find CRO/psychological patterns that should increase the baseline metric (e.g. via GoodUI.org).

That research is the foundation of your future work on the experiment concept, wireframes, and development. You know what the most important problem you want to solve is – nothing more, nothing less. 

If you follow any type of design process (e.g. this handbook), then that information should feed the process perfectly.

We believe that…

The second part summarizes the effects the experiment will have on business metrics. Its main goal is to provide you, or anyone else in the organization, a clear overview of the experiment’s tangible goal, main hypothesis statement, and the user segment that’s targeted. 

This part is referred to in the next section.

We’ll know by…

Last but not least, this part strictly relates to the documentation and statistical part of experimenting. You could easily use it as criteria or tags in your upcoming experiments/insights database.

The single most crucial action done here are the last two components – the clear KPIs you optimize and pre-test calculations we’ll discuss below.

Sample big hypothesis

If we put everything together, this is how the sample big hypothesis would look like for an experiment called “Optimizing Checkout Page #1”:

We know that in the last six months, the biggest drop-off (73%) in our business funnel happened on the third (of four) step – the checkout process. Based on HotJar recordings, the user research we did, and previous experiments, we know that users hesitate to pay the entire price at once. Also, we know that there’s something called the pain of paying in behavioral psychology that explains that friction.

We believe that improving the benefits section so it justifies paying the entire price better (V1) OR offering a partial payment (V2) for all users who reach the checkout page will result in a higher number of purchases.**

We’ll know by testing the “Optimizing Checkout Page #1” concept on all checkout pages and observing conversion rates for purchases for at least two weeks. We hope to see a minimal uplift of 7.32%.

**in this example we simplify things and don’t look at the revenue metrics, which would be affected because of the partial payments.

Approaching product experiments in this way results in significantly higher quality results and probability to have true uplifts.

Creating the experiment concept

The big hypothesis sets a framework for why, how, and what you want to test. The experiment concept describes what is within that framework. 

What’s included here?

• Scopes and changelogs: The list of variants and main changes that will be coded.
• Goals: List of goals (with KPIs) with a description of when and how the goal is triggered. This includes the micro-conversions too because we want to track not only the final conversion but also the CR changes within the entire funnel.
• Screenshots: Visual representations of each variant, with the changes.
• Links and materials: Wireframes, designs, links to reports, or experiment tools go here.
• Experiment setup (optional): It could contain any unusual experiment setup or general info, e.g. how the traffic was divided, why it was divided, etc.

After creating dozens of these in this form, I can say that it’s mainly done for the sake of documentation. However, it’s also very helpful when doing the experiment’s QA.

The first point, scopes, and changelogs address one issue that I heard from the audience during my recent ProductTank talk. Namely, how to know if we are trying to change too many things in the experiment at the same time. 

Listing the changes lets you (and other people) easily review the experiment concept, and make sure you’re able to explicitly check which change resulted in the uplift.

The importance of pre-test calculations

The last thing you need to do before you start working on the experiment is the pre-test calculations. Pre-test calculations tell you if you have the capability to get trustworthy results in a reasonable amount of time. Simply speaking, they tell you what the minimal sample size of visitors are and the minimal uplift you need to observe to have trustworthy results every week.

The more you experiment, the better intuition you’ll have on what the possible uplift you could achieve is. Pre-test calculations allow you to answer the following question: 

Based on the roadmap, can I justify the effort to devote the next X weeks in order to have the possibility of reaching Y% uplift?

That might be a deal-breaker because you want to grow as fast as possible. Remember, in most cases, small changes (like CTA on the button or its color) will not bring you high uplifts. 

If the pre-test calculations are not promising, you can do three things:

• Change the target segment to have higher volumes.
• Change the scope of the experiment. Bigger changes often lead to bigger downcasts/uplifts.
• Drop the experiment.

We’ll discuss how the pre-test calculations work in the next post.

Let’s get the party started

Doing the above will set you on the right track to have great results and to learn from your experiments. Whatever approach you have for the design and development process, it’s now time to put it into action. A final thought is to always do a thorough QA of the variants and experiment setup. 

Believe me, there’s nothing worse than finding a bug in the experiment or its setup after weeks of collecting data. That’s just a waste of money. To make it easier for you to ensure nothing is broken, here’s a checklist I use before starting any experiment.

When the experiment variants are coded and they pass the QA, it’s time to start the experiment and get the popcorn. The party has just begun.
Now it’s time to move to the second part of the article: Analyzing your experiment results.