Mainly because it's straightforward to do and doesn't require much effort or analysis. All you need to do is come up with two different price versions and split your visitors' flow into them equally. Whichever gets better conversions is the winner and as such, should remain on your website, pulling in the best customers in the tested segment.
Not.
A/B testing, as much as it works great for comparing landing pages or CTA (Call to Action) buttons, is not suitable for testing the pricing. And by "not good," I mean terrible to your business, which I will break down in detail below
Pricing consulting: it might be illegal
According to the EU's law, price discrimination is illegal when you charge customers differently just because they are of another nationality. The product's cost might be different, for example, due to other delivery costs or TAX rates, but you're not allowed to differ the price based on one's country of residence.
For example: if you're A/B testing the pricing of your project management product, and someone from France purchases it at a higher cost than a customer from Belgium, you might be fined if the one that paid more made a complaint to the European Consumer Centre.
Yes, you're conducting a random test, and such occurrences are not of your will, and yes, the chances that someone will spot the issue and then make an official complaint are low. But they're still here, which leaves the A/B testing in a grey area of legality.
Your traffic is not enough, most likely
Statistical Significance is a mathematical term that describes the reliability of a statistic. If the results of A/B tests are statistically significant, they represent an actual correlation between tested variations and their conversion rates.
In other words: they’re highly likely to be true.
However, for your split testing to be statistically significant, you need to conduct it on a particular group of people. The bigger the sample size you can get, the higher the significance. Understanding the behavior of the average customer is crucial, as it ensures your A/B test results are meaningful and applicable to your most common user segment.
Imagine you have monthly web traffic of 1000, and your conversion rate for purchasing a subscription is 1%. Your sample size of 10 users won’t give you any conclusive information at all. In this article by Michal Fiech, you will find instructions on calculating a minimum-size sample to make your test results trustworthy. Without sufficient data, it's difficult to accurately assess key metrics like customer acquisition cost, which are essential for effective pricing decisions.
And that’s not even it.
Most probably, you want to perform tests across different buyer personas. Not only do you need to identify them, but divide them into other test groups too. This leads to an even bigger required sample size, and most of the SaaS companies I speak with simply don’t have it.
A/B price test is suitable for websites with massive traffic and decent conversion rates. If you’re not in this group, it will give you false information.
You can put off your potential and existing customers
Due to the randomness of your traffic, the A/B test can lead to problematic customer service scenarios. If a sales specialist who got caught by your 10€ pricing is encouraging his or her boss to purchase software that to this person shows for 20€, you will most likely anchor them in the lower pricing point. It will be a nightmare converting them to higher pricing if it turns out to be the optimal one.
Now imagine another company where neither of these 2 was lucky enough to land on the cheaper plan.
They became regular users, and through word of mouth, they figured out that they could’ve paid two times less. I’m sure you wouldn’t like to be in their situation, and most likely, you’d raise questions. The more significant the price difference between plans, the harsher the questions will be.
To avoid any bad customer experiences, you need to be sure that they:
- use only one device to browse your website,
- don’t revisit it,
- are the only employees visiting the pricing plan,
- and never discuss software prices with their colleagues or anyone else.
Inconsistent or unclear pricing can confuse potential customers, leading to hesitation or loss of trust.
If you can’t be sure about even one of the above, then A/B tests will likely expose you to friction in the sales process.
You're overcomplicating things
If your pricing testing methods consist of 3 different plans, then the A/B test will double the number of buying possibilities and create a total of 6 segments.
On the marketing level, you’re making a different analytical thing to cover. From a sales perspective, you’re splitting your pricing policy, which is yet another thing to take care of by the sales team (just like the discount policy).
Cohort analysis is already complicated, and by multiplying it, you make it even harder. With A/B testing, you need to deal with two pricing plans that will affect your day-to-day sales and marketing efforts.
To avoid unnecessary confusion and operational challenges, it's important to keep your pricing structure simple.
You're limiting your business potential
The ultimate goal of A/B testing is to discover which of the two variables performs better. You’re revealing if the first one is more successful than the other in A and B’s duel.
This, by any means, doesn’t indicate that the winner is an optimal setup for your business. You can understand if A is better than B, but you don’t know if A is good at all. You’re limiting your global vision to nothing but multiple price points, which are just a part of complex pricing plans.
The perfect pricing point is found through surveying your existing and new customers and gathering their feedback. Leveraging customer insights from this feedback helps inform pricing decisions that better align with customer behavior and preferences. You need to understand what features they perceived value most and which they want to get rid of. And ultimately, what is their Willingness to Pay for the product?
If you limit your pricing strategy to nothing but A/B testing, you’re overlooking multiple factors that are significant in setting up an optimal pricing strategy. For example, understanding price sensitivity among different customer segments is crucial to ensure your pricing resonates with your target audience.
Or, as Vilgefortz, one of the sorcerers from The Witcher’s world, said: you mistake stars reflected in a pond for the night sky.⭐
A comprehensive approach that incorporates customer insights and market analysis is needed to truly optimize pricing, rather than relying solely on A/B testing.
What we have learned?
A/B testing is an excellent tool for comparing conversion rates of website visuals, such as CTA buttons or different web designs. It’s suitable for landing pages where significant amounts of people convert in short periods.
If your business is not in this stage yet, the A/B test won’t provide you with valuable conversion rate optimization insights. On the contrary, it will expose you to sales, marketing, customer service, and PR challenges.
To avoid those risks and to start working towards a suitable pricing strategy, I encourage you to read this article first. Adopting the right pricing strategies can help boost profitability by optimizing revenue and improving business margins. And in case of any questions, don’t hesitate to reach out to us via the contact form below. Businesses can also consider pricing consulting to develop more effective pricing strategies.
