Multivariate Testing: The Basics You Need

Multivariate Testing Defined

Multivariate testing (MVT testing) is a conversion rate optimization technique. The purpose of it is to maximize the conversion rate on your website.

In practice, CRO and multivariate testing aim to make your website better, so that you can get more customers out of your existing website traffic.

That translates to things like call to action buttons, attractive images, great copy, customer reviews, page speed, usability… Basically anything that makes your website visitors want to make a purchase, request a meeting, subscribe to your newsletter or whatever conversion event you are trying to achieve.

In multivariate testing, you can test multiple variables on multiple elements, and their combinations.

Basically, you choose what you want to test, set up a test, and the different page variations are shown to different visitors. In the end, you will get a report of which versions of the page performed best.

Of course, this technique has its limitations. You need to have enough traffic to be able to prove a statistical significance.

It’s important to note that multivariate testing is not just one testing method. 

It includes lots of types of testing methods, like full factorial testing, partial factorial testing, and Taguchi testing, of which some are online testing methods and some are offline methods… but let’s not dig too deep into the different methods. This is just FYI.

Benefits to Expect

Why should you do multivariate testing? 

Here are some benefits that you can expect from your multivariate tests. 

However, bear in mind that these benefits only apply when you use the test in appropriate situations, which are described later in this blog.

1. Better Conversion Rate

A successful multivariate test can result in higher conversion rates.

Or at least it can tell you that some elements have no effect on conversions whatsoever.

2. More Educated Guesses and Decisions

Once you know what works well, you are able to apply the knowledge to other contexts, too. 

When you build your new web page, you already know what is a good starting point.

You can also learn to look for answers elsewhere: if you have determined that your website conversions are not changing regardless of how much you play with multivariate tests, you can start looking for answers somewhere else.

3. No Need for Multiple Tests

In A/B testing, you would need to make lots of tests to reach the same results as with one single multivariate test.

4. Save Time and Money

See the above benefit.

Running just one test is faster and more cost effective than running multiple tests to find out the same test results.

Use Cases for Multivariate Testing

When to use multivariate testing?

Here are some use cases.

1. Testing Out Multiple Page Elements

Let’s say you own an ecommerce website.

You are still looking for the best style for your product page and considering different product images. So you take three different images of the product.

Additionally, you want to see which font is best for the product headline. You choose three versions.

In this MVT test, you get in total 9 different versions of your product page (3×3).

When the test is run, you will know which version is more likely to generate more sales.

Remember to take into account how much traffic you have on your website. 

Nine variables might be too much for one test, and you will not get meaningful results if you don’t have enough traffic.

2. When You Want to Make Radical Changes

Multivariate tests are best for larger scale changes. 

It doesn’t make sense to use them for smaller details: it takes too much time and is difficult to set up, and the achieved benefits are small compared to the amount of work.

Aim to choose those variables for your test that have the biggest impact on conversions, according to your hypothesis.

3. You Want to Know How Elements Interact with Each Other

You could run an A/B test on two versions of the page even if they have more than one variable. It’s not illegal or anything.

But you shouldn’t. I’ll tell you why.

Consider this: you have changed the CTA button, added a header image, and modified the headline for your A/B test. 

You expect version B to win you more customers, because the image is engaging and really cool. But you don’t pay enough attention to the fact that your new CTA button is not very visible.

After the test, you notice that the original version A performed better despite your expectations. Disappointedly, you give up the idea of the cool image and the other changes.

However, had you run a MVT test, where each combination of elements would be analyzed, you would have had a different result.

You would have noticed that the image would actually have a great positive impact on conversions if it was paired with your old CTA button.

Thus, the image and the CTA together would have a positive impact on conversions. You did not know that because you did not test all possible combinations.

That’s why a multivariate test is better when you want to change multiple things at once.

4. You Are an Advanced Professional

Multivariate testing is not the best choice for a rookie conversion rate optimizer.

The methodology is relatively complex, you need to carefully plan and set up the test, and analyzing the results is a demanding task.

When you master the skills of A/B testing, you can move on multivariate tests.

How to Do Multivariate Testing: 7 Steps

1. Analyze the Issue

If you are not getting the desired results from your website, you need to figure out why that is.

Sometimes you might be blind to your own creations: ask an outsider to give an opinion on why your visitors are not converting. 

Is the page functional? Does the visitor know what to do next? Does the page seem trustworthy?

You can find out these problems by collecting customer feedback, analyzing heatmaps, and checking metrics like bounce rate and time spent on page.

2. Define Goals

What is it that you want to improve exactly?

Are you aiming to get more sales directly from the website, or perhaps book more meetings with leads? Do you want to get more newsletter sign-ups or freemium users?

When you know what conversion event you are going to target, you will be able to analyze what factors play a role in the event.

Sometimes you could even focus on micro conversions, like clicks, form submissions, add to cart, views or downloads of a content piece, etc.

3. Create a Hypothesis

You surely have a hunch about what kind of changes would increase the conversion rate on your website, because you have analyzed the problem.

Write the hunch into a hypothesis or multiple hypotheses:

“Adding more professional looking product images will raise the conversion rate, because the products look more appealing to the customer.”

“Re-organizing the navigation bar will increase conversions because it will help customers find what they are looking for faster.”

“Showcasing customer reviews will make the visitors trust my brand more and increase the probability of purchase decisions.”

4. Create and Start the Test

Build your variables around the hypotheses you formulated previously, and set up the test using multivariate testing software.

See the list of multivariate testing tools below! They will help you with your mission.

Remember to carefully consider how many variables you want to test. Match them with your traffic to get an accurate result.

This is one of the situations where “go big or go home” doesn’t work.

Additionally, when you start getting some results and notice that there is a variable that is not performing as well as the others, you can let it go and keep testing with the other variables.

5. If Needed, Drive More Traffic

In case it looks like there is not enough traffic to your website, you should think about ways to drive more of it.

Traditional marketing techniques like social media marketing, paid advertising, PR and so on, will help with this task.

Remember that you need the visitors to be people who are genuinely interested in your company and would actually convert.

Sending a mass text to all your contacts saying “hey, please visit the website so that I can get a large enough sample for my test” is no good: it won’t generate you actual results.

6. Analyze Results

After the multivariate test is run, it’s time to see the outcome and find out if your hypothesis was correct.

Remember to take into account the statistical significance: was your sample large enough that you can draw conclusions? If the sample is too small, there is a risk that your results are merely a coincidence.

Sometimes you might find that your hypothesis was not correct. Maybe the elements that you tested were not decisive in the customer journey.

7. Implement Changes (Or Not)

If you get clear indication from the results that one version of the page performs much better, of course you should implement the changes.

However, sometimes you are just focusing on the wrong things. Maybe it is not your website that is the problem.

Maybe the product or service just doesn’t have that high demand, maybe your website is not responding to the intent of the visitors, maybe you are advertising for the wrong audience, or perhaps the prices are off.

Whatever the reason for low conversion rates might be, the way to find out is to ask for feedback from your customers and website visitors.

Good way to do so is to conduct feedback surveys like NPS.

Multivariate Testing Tools

Some great multivariate testing tools include:

1. VWO
2. Optimizely
3. Crazy Egg
4. Google Optimize 360
5. Trustmary Business

See our separate blog about best A/B testing tools for more in-depth reviews.

Conclusion

Multivariate testing enables you to test multiple variables at once, and learn how the different variations work together.

Special characteristics of multivariate testing are 

• You must have enough traffic to your page in order to get meaningful results
• It’s better suited for an advanced optimizer
• Not suitable for details, focus on elements with large impact
• It takes more time than A/B tests
• Methodology is rather complex

The world of MVT testing is confusing at first, but luckily there are many tools to help you on the way.

Further Reading

If you want to learn more about multivariate testing, see e.g.

• Multivariate Testing by VWO
• What is Multivariate Testing?
• When To Do Multivariate Tests Instead of A/B/n Tests

FAQ