Not everyone is in a position to perform testing.
Posted: Mon Dec 23, 2024 8:15 am
In this example, I want to detect a minimum 10% increase in my current 3% conversion rate, so I need 51,486 people to see variation A and 51,486 people to see variation B.
Now look at what happens if I want to detect, say, a 5% increase: I'll need a sample of 204,493 people for each variation (A and B).
CRO Tips: Sample Size Calculator
As you can see, running an A/B test requires quite a bit of traffic, and that number increases as the minimum detectable effect decreases. So the smaller the effect you want to detect, the more traffic you'll need.
This means two things:
People who can test don't want to waste their time and traffic on meaningless tests.
By ensuring you have a solid foundation before testing, you put yourself in the best possible position.
While all this research and preparation gives you an advantage, you should prioritize execution, whether in the form of a formal test, a quick fix, or an informal experiment. Johnathan Dane of KlientBoost explains:
Johnathan Dane, KlientBoost
“I wish people would be less romantic canada phone number sample about their primary area of expertise. You have to balance time and speed. Often, doing more research only yields marginal improvements in growth compared to executing, observing what happens, and then adjusting.”
“The next thing I wish was more people understood what prioritization is. Don’t fix the sink if the sink is broken.”
“Marketing is more holistic than people think, and there is always more growth to be discovered if you work from right to left, rather than the traditional left to right.”
"After conducting conversion rate research or a brainstorming session, you'll find yourself inundated with ideas for growth and optimization. The question then becomes, 'What should I do first?'"
That's where prioritization comes in. Don't let the ideas you're most excited about (or most biased toward) take over. Instead, use a prioritization method to decide what to do first. For example, the ICE (impact, confidence, ease) method is quite popular.
Johnathan also highlights the importance of staying close to the money. Instead of working towards the money (traffic, conversions, sales), work backwards, from the money back (sales, conversions, traffic).
Erin Bury of Eighty-Eight explains how this starts with proper performance measurement:
Erin Bury, Eighty-Eight
“Even if you’re not a data-driven marketer, you can still find ways to measure the impact of your efforts. Take influencer campaigns as an example: collaborating with influencers on Instagram or other social media can be a great way to get your product in front of a large, engaged audience, which in many cases can lead to sales.”
“But it’s important to track the impact of your influencer outreach to know which people are driving traffic and sales, and which ones aren’t delivering results. There are a couple of ways to do this: First, you can give influencers unique promo codes to track who is purchasing with that code.”
“Secondly, you can use UTM codes to create unique links, so if they’re sharing a link on Twitter, Facebook or other channels, you can see how many referral visits to your ecommerce site came from their unique links.”
“And finally, you can create an affiliate program so that when people click on an influencer’s link to your store, they get a share of the sales, and you can track those purchases. Even if you’re running campaigns that aren’t data-driven — say, a Facebook ad campaign — you can still be tracking metrics to measure the impact of your efforts and evaluate whether you should continue to invest in similar campaigns in the future.”
When deciding what to measure, work backward from money. Peep once said, “If you want to increase your conversion rate, make everything free.” Your conversion rate would skyrocket, but revenue would fall.
Make sure your metrics are accurate and as close to the money as possible. While it's easier, optimizing at the top of the conversion funnel (e.g. email open rates, number of page views) will have a weaker impact on revenue.
Resist the temptation to measure metrics out of vanity!
Speaking of human biases in testing, there is something called confirmation bias that Andre recommends you keep in mind:
Now look at what happens if I want to detect, say, a 5% increase: I'll need a sample of 204,493 people for each variation (A and B).
CRO Tips: Sample Size Calculator
As you can see, running an A/B test requires quite a bit of traffic, and that number increases as the minimum detectable effect decreases. So the smaller the effect you want to detect, the more traffic you'll need.
This means two things:
People who can test don't want to waste their time and traffic on meaningless tests.
By ensuring you have a solid foundation before testing, you put yourself in the best possible position.
While all this research and preparation gives you an advantage, you should prioritize execution, whether in the form of a formal test, a quick fix, or an informal experiment. Johnathan Dane of KlientBoost explains:
Johnathan Dane, KlientBoost
“I wish people would be less romantic canada phone number sample about their primary area of expertise. You have to balance time and speed. Often, doing more research only yields marginal improvements in growth compared to executing, observing what happens, and then adjusting.”
“The next thing I wish was more people understood what prioritization is. Don’t fix the sink if the sink is broken.”
“Marketing is more holistic than people think, and there is always more growth to be discovered if you work from right to left, rather than the traditional left to right.”
"After conducting conversion rate research or a brainstorming session, you'll find yourself inundated with ideas for growth and optimization. The question then becomes, 'What should I do first?'"
That's where prioritization comes in. Don't let the ideas you're most excited about (or most biased toward) take over. Instead, use a prioritization method to decide what to do first. For example, the ICE (impact, confidence, ease) method is quite popular.
Johnathan also highlights the importance of staying close to the money. Instead of working towards the money (traffic, conversions, sales), work backwards, from the money back (sales, conversions, traffic).
Erin Bury of Eighty-Eight explains how this starts with proper performance measurement:
Erin Bury, Eighty-Eight
“Even if you’re not a data-driven marketer, you can still find ways to measure the impact of your efforts. Take influencer campaigns as an example: collaborating with influencers on Instagram or other social media can be a great way to get your product in front of a large, engaged audience, which in many cases can lead to sales.”
“But it’s important to track the impact of your influencer outreach to know which people are driving traffic and sales, and which ones aren’t delivering results. There are a couple of ways to do this: First, you can give influencers unique promo codes to track who is purchasing with that code.”
“Secondly, you can use UTM codes to create unique links, so if they’re sharing a link on Twitter, Facebook or other channels, you can see how many referral visits to your ecommerce site came from their unique links.”
“And finally, you can create an affiliate program so that when people click on an influencer’s link to your store, they get a share of the sales, and you can track those purchases. Even if you’re running campaigns that aren’t data-driven — say, a Facebook ad campaign — you can still be tracking metrics to measure the impact of your efforts and evaluate whether you should continue to invest in similar campaigns in the future.”
When deciding what to measure, work backward from money. Peep once said, “If you want to increase your conversion rate, make everything free.” Your conversion rate would skyrocket, but revenue would fall.
Make sure your metrics are accurate and as close to the money as possible. While it's easier, optimizing at the top of the conversion funnel (e.g. email open rates, number of page views) will have a weaker impact on revenue.
Resist the temptation to measure metrics out of vanity!
Speaking of human biases in testing, there is something called confirmation bias that Andre recommends you keep in mind: