Experimentation in Product Management: The essential guide

Experimentation is one of the most important things that any Product team can do.

I don’t need to spend too much time on this subject, as so many people have written about experimentation before me. However, this would not be a Product Management blog if I didn’t cover it.

So this article will cover:

1. Why experimentation is crucial to Product Management.
2. What happens when you don’t run experiments
3. Finding the balance between experimentation and large-scale change
4. How to do experimentation well
5. How to create an amazing hypotheses

Why is experimentation crucial in Product Management?

Experimentation is the best way for any Product Management team to identify whether the changes they have made, really have their desired impact. This is crucial for 2 main reasons:

• You learn from your changes, and can make informed decisions about what to do next
• You can prove the value of your changes

Using experimentation to learn from your changes

Everything that we believe is an assumption until we prove otherwise. Even when we have a tonne of insight, and have conducted all of the Product Discovery – we cannot be 100% sure that the feature we build will have the impact that we desire until we launch it to our customers. Only then we really see if customers are using the feature, and if it is driving the behaviour that we expect.

If this is true – then that is great, our hypothesis and our assumptions were correct! We can use this insight to help us understand where we have impact and build on making changes in this space.

If this isn’t true, then we can question our assumptions and why we didn’t have the impact that we expected. This means that we can iterate and change our approach, so we do go back and gain the value that we desired.

I’ve written more about how to work through continuous optimisation, analyse your results and work out where to focus next.

Using experimentation to prove the value of your changes

Experimentation is the most powerful tool that you have to assess whether you’re focussing on the right areas, and showcase the value that you’re delivering back to the business.

Using experimentation allows you to see if you’re focussing on areas that actually have impact and value. If you aren’t – you can choose to focus on a different problem or area of the business. This means that you will increase the impact that you are having over time. You will do this far quicker than you would if you could not observe the impact of your change.

Without using evidence of impact, then a Product team is still very much a feature factory. The Product team is more likely to be working on projects, with agile delivery techniques. To really deliver value, you need to be able to see the impact you’ve had.

Experimentation is the best way to do this, as it provides a true reading on the changes you’ve made. Some teams still choose to use pre and post analysis. The harm is that other factors that might influence the observed change and the change has to be large for you to spot it.

You can also show this to be business, which allows you to have more influence on where you are choosing to focus and for them to be brought in to the success of the team.

What happens when you don’t run experiments in Product Management?

I am a huge believer in testing absolutely everything that you can, so that you can learn from every change that you make.

“The role of experimentation is to make better decisions, not perfect decisions…” Jonny Longden.

The issues that I have seen are when teams:

• Don’t test large website changes
• Test large changes, but haven’t validated hypotheses in the run up to this
• See experimentation as a separate stream of work to the day to day business activities
• Don’t believe that everything needs testing – they believe that they “know” that some of things are true / will be successful (there’s a reason why Booking.com has a 10% success rate, but industry-leading conversion rates). Even with all of that knowledge and history of testing, you can’t be sure on what will and won’t work.

This leaves teams with:

• A brand new site that doesn’t work as well as it used to, but they’re not sure why
• No real way to prove the value of the work they’re doing
• No certainty that what they’re doing is the right thing

The consequence is:

• More time and money spent on making changes
• A slower learning curve
• Less benefits delivered to your customers
• Competitors will learn faster and deliver better experiences
• A lower return on investment for the Product team

Even if you have a business model where you are happy to go live with a change, even if it is a inconclusive experiment, you will still benefit from experimentation. The benefits of testing are that no matter what, you will learn from what you do. These learnings to allow you to continue to grow at a faster pace.

Finding the balance between experimentation and large-scale change in Product Management

I’ve also experienced the opposite effect, where a business tests everything.

If this is true, and the metric being measured is too narrow, then it can be difficult for a business to take a leap of faith and try something brand new.

Instances where a Product team might need to make bolder changes are if they are trying to generate a completely new brand, with a completely new audience.

If this is the case, then the success metrics for this kind of launch should be focussed on % mix of new target audience, or visibility in this space. AB testing is more difficult, as you don’t have the right baseline to test against.

This might mean taking a hit to overall profits whilst the business transitions.

Therefore – the balance and beauty of experimentation lies in:

• Choosing the right metric(s) to allow you to make the correct decisions and deliver long-term success
• Testing everything (to validate assumptions and act as a safety net – this doesn’t mean that you can’t switch on ‘failed’ experiments, if you chose to do so for another reason – it just allows you to have the visibility to be aware of this)
• Take a leap of faith – with a safety net around it (non-inferiority testing) to allow you to then optimise your way up from there

PS. Here’s some more thought-leadership on how experimentation can work with innovation.

Hopefully you’re now convinced that you need to focus on experimentation, let’s talk about how to do experimentation well.

How to do experimentation well in Product Management

If you’re using experimentation, it is important to do this correctly. Not doing so will mean that you potentially find false positives and think that you are having a better impact than you actually are. A lot of teams are desperate to prove their success, and therefore often fall into a few traps.

Some common trpas are:

• Observing the experimentation too early and drawing false conslusions
• Finding positive metrics in areas other than they desired

Any experimentation purists would tell you that both of those factors are very poor practise. As a former Product Manager, I would say that both are entirely natural and can still allow you to find strong learnings (don’t tell anyone that I said this!).

The 4 rules of experimentation in Product Management

The 4 rules that you do need to stick by to experiment successfully are:

• Choosing a confidence interval and sticking to this
• Allowing an experiment to run for at least two weeks
• Identifying your hypothesis before you create your experiment
• Creating clear primary and secondary success metrics

For all of your experiments, you should select a confidence interval. This basically signals your willingness to take risks. If you choose 90% instead of 95%, you are indicating that you are hoppy to be less certain that your change is definitely a positive result. The interval you choose entirely depends on your business, and your risk appetite. Standard industry practise is 95%.

Allowing an experiment to run for 2 weeks allows you to ensure that there are no anomolies in behavioural patterns. For example, if a holiday or particular event changes customer behaviour, it should be evened out over the course of 2 weeks.

Creating a strong hypothesis and success metrics

So many companies experiment. That isn’t the problem. The problem is whether they’re actually learning through experimentation, or just looking for any signal that looks good to prove their success.

The issue with this?

• You could be looking at false positives
• You’re not really delivering the value that you think you are
• Your learnings become confusing – you continue working on areas that haven’t actually delivered value, as opposed to pivoting and working on something that might.

Why do companies end up in this position? Because they don’t craft effective hypotheses.

I’ve seen so many teams go to deliver a feature and then question what they’re measuring. They assign some metrics to this and hope that one of the metrics moves. It often does – which is a great sign (see continuous optimisation!). But… if it wasn’t the metric you intended to move, have you really achieved your goals? Or have you just learnt something that you can use to help you to continue to reevaluate and grow?

How to write a good hypothesis

It is:

• Linked to the objectives that you want to achieve (your outcomes)
• Crafted before you fully design features
• Measurable

My favourite format to use – that focuses on quality, but doesn’t make the process too laborious – so it is easy to adopt in teams, is:

Based on [evidence].

We believe [X] will encourage [these users] to [behaviour].

We will know this when we see [effect] happen to [metric].

This incorporates all of the important factors that you need:

• Why you believe in your idea (this helps you to create a feature that should work, based on insight)
• What your feature is
• Who it is for
• The behaviour you want to change
• The primary metric that you will use to measure this

For example:

Based on the questions that I have been asked by various Product Managers.

I believe that creating this article will encourage Product Managers that want to learn more about experimentation, to implement better experimentation practises in their business.

I will know this is true when 5 PMs tell me that this article helped them to make a change in their business.

If 5 PMs do not tell me this, then either my article is sh*t, or .. my hypothesis that an article is what my audience need is wrong, and I should try a different tactic, orrrr … PMs actually don’t care that much about crafting better hypotheses in reality.

Why is this helpful?

• I have a true gage on what success looks like to me – 5 PMs. In a business context, this might be the minimum shift you’d need to see to make something worth dev effort, or to cover costs – or your usual success ie. a 2% shift.
• I am very clear on who I’m targeting. Any positive feedback from anyone outside of my target group is interesting, but doesn’t mean I achieved my goal (the initial problem that I was trying to solve)
• I am clear on why I’m writing this – based on questions that I have been asked. This keeps me focussed on creating something that is actually useful.

How to identify good success metrics

If you are already using outcomes in your Product Organisation, then you will know overall which metrics you want to drive. Your experiment might not be able to hit this leading indicator, but you should be able to target a metric that should contibute to this.

Choosing your success criteria should be based on the:

• Outcome that you want to drive
• The audience thatyou are driving this for
• The area of product that you can influence
• The evidence that you have that inspired your hypotheses

With this combination of factors, there will be some clear success criteria that you could measure.

Choosing one will depend on the nature of your experiment. I have had examples in the past, where I had 3 primary metrics that I could measure. The way that we were able to select one, was based on the one that would experience the most impact from the area of the funnel that we were testing on. This success criteria would be our best way to measure a conclusive change in the shortest time-period.

Choosing the right primary and secondary metrics for your experiment might not be 100% obvious, but with all of the above information – and by working with your analyst, you will be able to identify what change you really want o see and how best to measure it.

Conslusion

Experimentation is crucial to the success of Product Management. It allows you to learn quickly, identify where you need to optimise, and prove where you are delivering plenty of value. Experimetnation is also a great tool to help you manage innovation and large scale change successfully.

To gain the value from experimentation, you need to doexperimentation well. To do this, you need to craft greathypotheses, choose appropriate success metrics and follow the 4 rules of experimentation.

If you take on all of this advice, I know that you will succeedin experiments.

If you would like to read more about experimentation, from AB testing basics, to how to do this continuously well and make it part of your culture – check out these links.

The basics of experimentation:

Absolute basics of AB testing.

A complete experimentation overview.

Analysing AB tests.

Analysing your experiments:

Taking into account your appetite for risk

Things to be aware of when testing:

Experimentation pitfalls.

Making experimentation part of your culture:

Product-led experimentation.

Making experimentation part of your culture.

Combining qualitative research with experimentation.