bayesian vs frequentist a b testing

Optimize landing pages with:

  • a/b tested testimonials and reviews
  • a/b tested lead generation widgets

Sign up free

Let me guess: you have been roaming around the internet for information on A/B testing, but you only come across highly technical and difficult explanations.

I feel your pain! I’m not the most technical or mathematical person myself.

But worry not, this blog will explain the principles of bayesian and frequentist A/B testing in layman terms. (At least try: our product owner thinks there is no “for dummies” explanation about this topic!😅)

After reading this, you won’t be an expert in these methods, but at least you know what they are.

If you are looking for more specific methods and instructions for how to calculate probabilities yourself: better luck next time!

What Is A/B Testing?

In case you know nothing about A/B testing, let’s have a brief overview.

Generally speaking, A/B testing means that you use statistical data to figure out which elements have the greatest effect on your website conversion rates (or advertisements, newsletters, etc).

Basically, you: 

  1. create two (or multiple) versions of your desired web page, 
  2. show these versions to your website visitors (there are different ways to do this), and 
  3. after the test is done, you analyze the results and see which version performed or is likely to perform better (=which version generates more conversions).

This way you don’t have to base your marketing decisions on a “hunch” or feeling, but you have real life data on what truly works.

a/b test principle

You can analyze the testing data following different approaches, like Bayesian statistics or Frequentist statistics.

How do bayesian and frequentist approaches differ from each other, then?

Basically, they have different interpretations for probability. We will have a brief look at the differences soon.

However, we don’t go into too much detail about the statistics. If you want to learn more about the exact methods, see the further reading at the end of the article!

What Kind of Things Do People Measure with A/B Testing?

The things you test don’t have to be grand.

In fact, the smallest details can sometimes have a great effect on the customer experience.

I’m talking about things like placement of a CTA button, or which presented element has a greater effect on people’s behavior.

For example, if you showcase reviews or testimonials on your website, you can try out different options and perhaps learn something surprising.

a/b test reviews

Furthermore, it would not make sense to test on too many variables at the same time. 

If you have 5 different things going on, how can you pinpoint which ones truly affect the conversions?

It’s best to test small things one by one to ensure that you get accurate results.

Basic Idea of Frequentist Approach to A/B Testing

The frequentist approach is the more traditional one of the two.

Remember, you have two versions of the website: variable A and variable B.

You suppose that one of the following hypotheses is true:

  1. There is no difference between A and B (=null hypothesis)
  2. There is a difference between A and B

Your job (or the software tools’ job) is to test out which one is true. Moreover, you need enough evidence to prove that the null hypothesis is not true.

hypothesis definition

To do this, you choose a sample size (=number of visitors) that shall see each version of the website.

Once the sample size is full, you stop the test and can start analyzing the results.

The problem with the test is that the variants will likely not have the exact same conversion rate even if they were identical. There are bound to be some differences due to coincidence.

Because of this, you need to know the statistical significance of the finding, which is the p-value. You have to prove that the result is not just a coincidence.

When the p-value is adequate, you can assume that the result is statistically significant and somewhat accurate.

To find out how to calculate the p-value, refer to e.g. this article.

frequentist a/b testing method

Basic Idea of Bayesian Approach to A/B Testing

The ideology behind Bayesian thinking is that you update your beliefs whenever you acquire new information. The statistical method is also based on this ideology.

Despite being newer, the Bayesian approach has become the standard in the digital marketing industry.

The basic idea of Bayesian statistics is that it is calculating probabilities rather than testing hypotheses.

It calculates the probability distribution of each variant based on prior information. The prior knowledge comes from e.g. what is the average conversion rate for a website.

Basically, you get an answer to the question “How likely will A be better than B” or vice versa.

The probability estimate is getting more and more accurate when you collect more observed data.

The math behind the Bayesian method is complex, so I won’t even go there.

bayesian a/b testing method

Which One Should You Choose?

In terms of results, there isn’t much difference between the approaches. If one variant performs better than the other, both methods will reveal it.

However, the processes look a little different.

Experts seem to be moving towards the Bayesian approach, as it is more flexible and faster. It is also the method that most A/B testing software tools use.

If you consider what we just learned about the two approaches, it makes sense.

In the frequentist approach, you must wait until a certain data sample is fulfilled. 

In Bayesian analysis, you can get estimates earlier. You just need to decide when is the best time to take the risk and implement changes.

TLDR: What’s The Difference and What Does It Mean for Me?

The differences between Bayesian and Frequentist methods are: 

  • In the Frequentist approach, you test hypotheses and get a fixed point estimate (=what is the “true” conversion rate in your test).
  • Bayesian statistics assign probabilities for the hypotheses, not a fixed number (=how likely is one option better than the other).
  • Frequentist testing takes more time than Bayesian estimation.

But what does this mean for your business? Why should you care?

Digital marketing experts seem to think that Bayesian methods are better and allow more agile reactions.

However, the results are ultimately similar regardless of method. It’s not like Frequentist analysis says that A is better, while Bayesian makes the opposite conclusion.

So, if all of this is going over your head and you don’t have a strong ideological preference for either option… Maybe this is not the most important thing that you should be considering.

Moreover, when you implement A/B testing software, you don’t have to think about these things. 

Maybe you want to choose a tool that uses one or the other of these approaches, but other than that, you probably should focus on something else.

Further Reading

If this guide for dummies was not enough for you, here are some resources that might help you further:

In case you got bitten by a statistics bug, here are some heavier resources:


What is A/B testing?

In A/B testing, two variables are compared to each other with statistical analysis. In digital marketing, it usually means testing out which version of the website or advertisement generates more conversions.

What are frequentist and bayesian methods?

Frequentist and Bayesian approaches are methods in statistical analysis. They can be applied to e.g. analyzing the results of an A/B test.

What are frequentist statistical methods?What are frequentist statistical methods?

In A/B testing, Frequentist statistical methods provide an answer to the question “Which website has the better conversion rate?”. First, you define variables and a sample size. After that you examine their differences and use the p-value to determine the statistical significance of your findings. 

What is the Bayesian method?

The Bayesian method focuses on finding out which variable of your A/B test is likely to perform better. The more data you collect, the more accurate the estimation becomes. The Bayesian method doesn’t give an exact answer, but rather a range in which the variable will perform.

Which is better, frequentist or bayesian approach?

Both methods will tell you essentially the same things. However, experts in the field root for Bayesian methods these days, as they are more flexible and provide estimations faster.

Improve conversion rates by 20% with Trustmary!

Sign up for free and start your first experiments!