What is A/B Testing? A Beginner’s Guide

No matter how much research you do, not every marketing campaign will get positive results.

That’s why A/B testing is a fantastic method to calculate the best online promotional and marketing strategies for your business.

It can be used to test everything from website copy to sales emails. This allows you to find the best-performing version of your campaign before spending your entire budget on marketing materials that don’t work. While A/B testing can be time-consuming, its advantages are enough to offset the time investment.

All in all, well-planned A/B tests can make a huge difference in the effectiveness of your marketing efforts. Narrowing down and combining the most effective elements of a promotion can create a higher return on investment, lower risk of failure, and above all: a stronger marketing plan.

What Is A/B Testing and Why Does it Matter?

A/B testing is a marketing strategy that pits two different versions of a website, advert, email, popup, or landing page against each other to see which is most effective.

For example, you might test two different popups (to see which drives more webinar sign-ups) or two different Google Ads (to see which drives more purchases). This provides key insights on where and how to invest your marketing budget and gives you the courage to take potentially risky moves.

On my own site, I A/B tested my popup to find out what encouraged users to engage with my brand.

Over time, we found offering a free website analysis (which provides tons of value to our target audience) was the most effective way to establish expertise and show visitors our value. We were able to compare clicks and conversion rates to see which header would be more impactful to keeping users on our site.

How Does A/B Testing Work?

A/B testing works by randomly showing two versions of the same asset (ad, website, pop-up, offer, etc.) to different users. The random part is important because this provides more accurate information without skewing the results.

One version is the “control” group, or the version already in use. The second version changes a single element. You can change multiple elements, but it does make it harder to tell what change made the difference. This is called multivariate testing (more on this later).

For example, you might show half of your website visitors a blue “buy now” button and the other half a red “buy now” button. After a certain period of time (generally at least two weeks), you’d compare conversion rates to see which color button resulted in more purchases.

Most marketers use a tool to create and display the different versions—we’ll cover A/B testing tools in a later section.

Why Is A/B Testing Important?

Accurate A/B tests can make a huge difference in your return on investment. By using controlled tests and gathering empirical data, you can figure out exactly which marketing strategies work best for your company and your product.

If there’s a possibility that one variation is working two, three, or even four times better than another without putting large capital at risk, it’s careless to run a promotion without testing first.

When done consistently, testing can improve your results substantially. If you know what works and what doesn’t (and have evidence to back it up) it’s easier to make decisions and craft more effective marketing strategies in the long run.

Here are a few other benefits to running regular A/B tests on your website and marketing materials:

• They help you understand your target audience: When you see what types of emails, headlines, and other features your audience responds to, you gain insight into who your audience is and what they want.
• Higher conversion rates: A/B testing is the single most effective way to increase conversion rates. Knowing what works and what doesn’t gives you actionable data that can help you streamline the conversion process.
• Stay on top of changing trends: It’s hard to predict what type of content, images, or other features people will respond to. Testing regularly helps you stay ahead of changing consumer behavior.
• Reduce bounce rates: When site visitors see content they like, they stay on your site longer. Testing to find the type of content and marketing materials your users like will help you create a better site — and one that users want to stay on.

Ultimately, you’ll be regaining control of your marketing strategies. No more closing your eyes, pressing the “send” button, and merely hoping your customers will respond.

How Do You Plan an A/B Test?

The first thing to do when planning an A/B test is figure out what you want to test. Are you running an on-site test, or an off-site test?

If you’re running an on-site test, you’ll want to think of all the sales-related pieces of your website, and then figure out which elements to split test.

You might test:

• headlines
• calls to action text
• calls to action location
• pop ups
• featured images
• copy
• the number of fields in a form

With off-site tests, you’re probably testing either an ad or a sales email. Testing ad copy to see which ad drives more conversions can help you focus your advertising efforts. Once you know your ad is converting as well as possible, it’s easier to justify spending more money on it.

The same goes for emails. Send out two versions to your list (randomly selecting which half gets which email), and then track which one converts better. With emails, you can adapt your structure, email subject line, images used, or even your offers.

Knowing what your audience responds best to allows you to write more effective emails in the long run. Once you know what marketing material you want to A/B test, make a list of all the variables. If you’ve decided to test your call to action, you might test:

• the location
• the exact text used
• the button color or surrounding space

A/B testing is a whole process, and it’s common for multiple split tests to be performed prior to making your final decision.

A/B Testing Checklist to Get Started

Before you start split testing, make sure you have a clear idea of the results you’re looking for. You should already know your baseline result, which is the results you’re currently getting. You want to test options A and B against each other, but you also want to know that whichever one does better in the test is also doing better than your current results.

Alternatively, you can use A as your control (leaving whatever you’re currently using) and then use something new for B.

Tests need to be run simultaneously to account for variations in timing. You can’t test one variation today and the other one tomorrow because you can’t factor in any variables that might have changed between now and then. (For example, a new Facebook campaign or a blog post going live.)

Instead, you need to split the traffic seeing your variations at the same time.

Here’s an A/B testing checklist to review before running your first test:

1. Decide what feature you want to test.
2. Create two versions of the same ad, landing page, app, etc.
3. Decide how long your test will run. I suggest at least two weeks, but it may be longer or slightly shorter depending on your traffic and industry.
4. Choose a testing tool to help you run your test (more on that later).
5. Launch!
6. After a couple of weeks, take a look at the results. Which version won?
7. Rinse and repeat. A/B testing is most effective when done continually.

Top Elements to Test Using A/B Testing

You can test virtually anything in your marketing materials or on your website: headlines, CTAs, body copy, images, navigation bar placement, etc. If you can change it, you can test it.

That doesn’t mean you should spend months testing every little thing. Instead, focus on changes most likely to have a big impact on traffic and conversions.

On your website, this likely includes:

• the headline
• your CTA
• any graphic you use in direct correlation to your sales efforts
• the sales copy or product descriptions
• featured image
• button size and placement

In an email, you might test the title, images, links, CTAs, or segmenting options. In a paid ad, especially a text ad (like a search ad), you have fewer things to change, so you might test the main headline, the offer, image, or targeting.

Testing different offers is crucial. Just make sure that each person is always offered the same promotion. For example, if a free gift is offered to group A, and a discount is offered to group B, then you want to make sure that group A always contains the same visitors, as does group B.

You can also test the full path of conversion. For example, you might test newsletter A with landing page A, and newsletter B with landing page B. Later, you may want to test newsletter A with landing page B, and vice versa.

This can give you a better idea of what is working, especially if you’re getting mixed results or if the results are very close. Here are a few other tests you can run.

Practical A/B Testing Examples to Inspire Your Next Campaign

Now that we’ve covered what A/B testing is, what you can test, and how to do it, let’s look at a few examples. These should help highlight the power of A/B testing—and what you could be missing if you aren’t using them.

GRENE Tested Horizontal Layout on Category Pages

GRENE, an online retailer, ran an A/B test to find a way to make it easier for users to find the products they were looking for. In the original version (left), the product took up the entire page on mobile devices. Users were finding it difficult to scroll through the different options.

The variant (right), reduced white space and allowed users to view several products and easily scroll through the available options.

Results: By changing the layout of category pages, GRENE saw a 15 percent increase in product box clicks, a 16 percent increase in conversions, and a 10 percent increase in visits to the thank you page, which indicates users made a purchase.

WallMonkeys Increased Conversion Rates by Replacing a Slider With a Search Bar

WallMonkeys, an online wall decal website, wanted to improve conversion rates and customer experience. Using CrazyEgg’s heatmap, they were able to see where most customers looked first.

Armed with that information, they decided to swap the slider featured image (top image) for a search bar (bottom image.)

Results: By swapping out featured images and moving the search bar to the middle of the page (based on heatmap data), they were able to increase their conversion rate by 550 percent.

Unbounce Tested a Tweet Vs. Email Opt-in

Unbounce was looking for ways to increase landing page opt-ins. While most businesses ask for an email address, Unbounce decided to see if users would prefer tweeting about a product instead.

So, they compared this opt-in page, which asked for an email address:

This version lets users download the same course by sending out a tweet.

How did the changes impact sign-ups?

Results: Unbounce found users preferred to give out an email address to download the course. The email version had a 24 percent higher conversion rate than the tweet version. The results aren’t surprising (most people are used to giving out an email address, after all), but testing gave Unbounce the confidence that their landing page was going in the right direction.

How Much Time Does A/B Testing Take?

A/B testing is not an overnight project. Depending on the amount of traffic you get, you might want to run tests for anywhere from a few days to a couple of weeks. Remember, you only want to run one test at a time for the most accurate results.

Running a test for an insufficient amount of time can skew the results, as you don’t get a large enough group of visitors to be statistically accurate. Running a test for too long can also give skewed results, though, since there are more variables you can’t control over a longer period.

Make sure that you stay abreast of anything that might affect your test results, so that you can account for statistical anomalies in your results. When in doubt, run the test again.

Considering the impact A/B testing can have on your bottom line, it’s worth taking a few weeks to properly conduct tests. Test one variable at a time, and give each test sufficient time to run.

Can I Test More than One Thing at a Time?

There are two approaches to this question. Say you just want to test your headline, but you have three possible variations. In that case, running a single test and splitting your visitors (or recipients in the case of an email) into three groups instead of two is reasonable, and would likely still be considered an A/B test.

This is more efficient than running three separate tests (A vs. B, B vs. C, and A vs. C). You may want to give your test an extra couple of days to run, so you have enough results to see what actually works.

Testing more than one thing at a time, such as headlines and calls to action, is called a multi-variate test, and is more complicated to run. There are plenty of resources out there for multi-variate testing.

You’ll also need to consider how your systems can handle split tests as well as have staff on hand able to analyze multiple results and compile the data into digestible amounts.

Multivariate testing puts a lot more on your plate at once: but it shouldn’t necessarily be avoided. If you have the right procedures in place to handle the extra work load, then go ahead – but if you want a more simplistic approach: one A/B test at a time is just fine.

How to Analyze A/B Testing Data

After your A/B test, you’ll have a pile of data. How do you know which version won? Sometimes the winner will be quite clear. For example, if one version of a landing page results in 50 percent more email sign-ups, you know who won without analyzing much data.

Other times, it’s not so clear. Here’s how to make sure you know which version truly won:

1. Make sure you have enough data: The best way to tell which variation will perform in the long run is to ensure you have around two weeks of data (at least 30 conversions).
2. Use an A/B testing significance calculator: Some have a built-in A/B testing significance calculator, or you can use my free tool here. Just add your visitor and conversion numbers to see how much the variation increased sales.

Look beyond the obvious metrics: Not all metrics are created equal. I generally advise looking at conversion rates and traffic. However, some businesses might want to pay attention to other metrics, like average order size. For example, changing your “buy” button to blue results in more conversions, but those customers spend significantly less per order, so you’ll want to keep digging.

Best A/B Testing Tools to Try

If A/B testing sounds complicated, you’re not alone. Plenty of marketers and business owners avoid A/B testing because it feels like too much work or they’re worried they’ll do something wrong. Hopefully, the tips above will help you feel confident you can do it correctly. Now, let’s talk about tools you can use to do A/B testing.

The tool you use will depend on the features you want to test. For example, if you want to test email headlines, your email provider likely offers this tool (both MailChimp and Constant Contact offer this). Facebook ads also offer this feature.

There are also several cheap or free tools that can test website elements and help you understand which variant is most effective.

Free A/B Testing Significance Calculator

If you’ve been wondering how a design or web copy change impacted your sales, then I’ve designed a tool to help. My calculator allows you to enter your visitor and conversion numbers, and then works out if a variation increased your sales, and by how much.

FAQs

What is A/B Testing and Why Does It Matter?

A/B testing is a marketing strategy that pits two different versions of a website, ad, email, popup, or landing page against each other to see which is most effective. It’s one of the most effective ways to increase conversion rates.

How Do You Plan an A/B Test?

Decide what to test, create two versions, decide on how long to run the test, choose a tool, then see what works!

What Should I A/B Test?

Any part of a paid ad, website, or marketing material including (but not limited to) pop-ups, emails, landing pages, and featured images.

How Much Time Does A/B Testing Take?

Most tests should be run for at least two weeks, but A/B testing should be continual.

Can I Test More Than One Thing At A Time?

Yes, in some cases. In general, it is best to stick to two versions of the same asset.

What A/B Testing Tools Should I Use?

Google’s Optimize is a free, powerful A/B testing tool. Your email platform, landing page tools, or website plugins may also offer this feature. For paid tools, consider Optimizely.

Conclusion

A/B testing is a marketer’s best friend. It allows you to see, for example, what ads drive the most conversions, what offers your audience responds to, or what blog headlines drive the most traffic.

There are a variety of tools you can use to get started, including Google Optimize (which is free!) and Optimizely.

If you’re looking to get started with A/B testing, you can start by learning how to A/B test in Google Analytics. Remember: A/B testing is a fantastic tool that should be used by all marketers.

Have you tried A/B testing? If not, what is holding you back?