Bayesian vs Frequentist A/B Testing: Guide for Dummies

What Is A/B Testing?

In case you know nothing about A/B testing, let’s have a brief overview.

Generally speaking, A/B testing means that you use statistical data to figure out which elements have the greatest effect on your website conversion rates (or advertisements, newsletters, etc).

Basically, you: 

1. create two (or multiple) versions of your desired web page, 
2. show these versions to your website visitors (there are different ways to do this), and 
3. after the test is done, you analyze the results and see which version performed or is likely to perform better (=which version generates more conversions).

This way you don’t have to base your marketing decisions on a “hunch” or feeling, but you have real life data on what truly works.

You can analyze the testing data following different approaches, like Bayesian statistics or Frequentist statistics.

How do bayesian and frequentist approaches differ from each other, then?

Basically, they have different interpretations for probability. We will have a brief look at the differences soon.

However, we don’t go into too much detail about the statistics. If you want to learn more about the exact methods, see the further reading at the end of the article!

What Kind of Things Do People Measure with A/B Testing?

The things you test don’t have to be grand.

In fact, the smallest details can sometimes have a great effect on the customer experience.

I’m talking about things like placement of a CTA button, or which presented element has a greater effect on people’s behavior.

For example, if you showcase reviews or testimonials on your website, you can try out different options and perhaps learn something surprising.

Furthermore, it would not make sense to test on too many variables at the same time. 

If you have 5 different things going on, how can you pinpoint which ones truly affect the conversions?

It’s best to test small things one by one to ensure that you get accurate results.

Basic Idea of Frequentist Approach to A/B Testing

The frequentist approach is the more traditional one of the two.

Remember, you have two versions of the website: variable A and variable B.

You suppose that one of the following hypotheses is true:

1. There is no difference between A and B (=null hypothesis)
2. There is a difference between A and B

Your job (or the software tools’ job) is to test out which one is true. Moreover, you need enough evidence to prove that the null hypothesis is not true.

To do this, you choose a sample size (=number of visitors) that shall see each version of the website.

Once the sample size is full, you stop the test and can start analyzing the results.

The problem with the test is that the variants will likely not have the exact same conversion rate even if they were identical. There are bound to be some differences due to coincidence.

Because of this, you need to know the statistical significance of the finding, which is the p-value. You have to prove that the result is not just a coincidence.

When the p-value is adequate, you can assume that the result is statistically significant and somewhat accurate.

To find out how to calculate the p-value, refer to e.g. this article.

Basic Idea of Bayesian Approach to A/B Testing

The ideology behind Bayesian thinking is that you update your beliefs whenever you acquire new information. The statistical method is also based on this ideology.

Despite being newer, the Bayesian approach has become the standard in the digital marketing industry.

The basic idea of Bayesian statistics is that it is calculating probabilities rather than testing hypotheses.

It calculates the probability distribution of each variant based on prior information. The prior knowledge comes from e.g. what is the average conversion rate for a website.

Basically, you get an answer to the question “How likely will A be better than B” or vice versa.

The probability estimate is getting more and more accurate when you collect more observed data.

The math behind the Bayesian method is complex, so I won’t even go there.

Which One Should You Choose?

In terms of results, there isn’t much difference between the approaches. If one variant performs better than the other, both methods will reveal it.

However, the processes look a little different.

Experts seem to be moving towards the Bayesian approach, as it is more flexible and faster. It is also the method that most A/B testing software tools use.

If you consider what we just learned about the two approaches, it makes sense.

In the frequentist approach, you must wait until a certain data sample is fulfilled. 

In Bayesian analysis, you can get estimates earlier. You just need to decide when is the best time to take the risk and implement changes.

TLDR: What’s The Difference and What Does It Mean for Me?

The differences between Bayesian and Frequentist methods are: 

• In the Frequentist approach, you test hypotheses and get a fixed point estimate (=what is the “true” conversion rate in your test).
• Bayesian statistics assign probabilities for the hypotheses, not a fixed number (=how likely is one option better than the other).
• Frequentist testing takes more time than Bayesian estimation.

But what does this mean for your business? Why should you care?

Digital marketing experts seem to think that Bayesian methods are better and allow more agile reactions.

However, the results are ultimately similar regardless of method. It’s not like Frequentist analysis says that A is better, while Bayesian makes the opposite conclusion.

So, if all of this is going over your head and you don’t have a strong ideological preference for either option… Maybe this is not the most important thing that you should be considering.

Moreover, when you implement A/B testing software, you don’t have to think about these things. 

Maybe you want to choose a tool that uses one or the other of these approaches, but other than that, you probably should focus on something else.

Further Reading

If this guide for dummies was not enough for you, here are some resources that might help you further:

• Bayesian vs. Frequentist A/B Testing: What’s the Difference?
• Frequentist vs. Bayesian approach in A/B testing
• How to run better and more intuitive A/B tests using Bayesian statistics

In case you got bitten by a statistics bug, here are some heavier resources:

• Gronau, Q. F., Raj, K. N., & Wagenmakers, E. J. (2019). Informed Bayesian inference for the A/B test. https://arxiv.org/pdf/1905.02068.pdf 
• Mayo, D. G., & Cox, D. R. (2006). Frequentist statistics as a theory of inductive inference. In Optimality (pp. 77-97). Institute of Mathematical Statistics. https://doi.org/10.1214/074921706000000400 
• M. J. Bayarri. J. O. Berger. “The Interplay of Bayesian and Frequentist Analysis.” Statist. Sci. 19 (1) 58 – 80, February 2004. https://doi.org/10.1214/088342304000000116

FAQ