Being able to test nearly everything isn’t a license to do so. As we’ve said before, you can’t just randomly start building and throwing variations at your customers. Just like laying a foundation for a home, figuring out what to test is the first and most important step.
Let’s keep with the political candidate from the previous topic whose visitors weren’t signing up for his newsletter and look at how he might choose what to test. Some solutions to increasing signups might be:
Knowing a few examples of what to test is helpful, but how do you decide where to start? Two things to be aware of when considering a solution are:
All tests need to be subject to the same variables
A/B tests need to be subject to the same variables in order to avoid issues stemming from seasonal fluctuations or random events outside of your control. Just as if you were reproducing a science experiment, the variations need to be compared under similar conditions and in a single test. In order to directly compare the performance of variations against each other, you need to test at the same time. Testing long enough and on sufficient traffic will ensure that fluctuations won’t skew the results.
Taking your time is crucial. Tests require an adequate run to really determine what is and is not working. After a sufficient amount of traffic has rolled in (figuring out just how much is needed is a skill itself), the data becomes clearer and less susceptible to noise (GA experiments will even stop the test once they’ve reached certainty!).
A/B testing requires patience and repetition. If you don’t obtain significant results the first few times, keep testing. Focus on creating a large number of smaller tests and refining your testing each time with the things you’ve learned. Did your hypothesis hold up or does it need tweaking? What did the variations that performed better have in common?
The best thing about A/B testing? It always produces a result. You can run a test for days backed with a great hypothesis and backed by concrete data, and it could still return next to nothing in terms of actionable data. However, that lack of a result will teach you what did not work, and give you more insight into your customer’s behaviour.
This information can now be used to change your assumptions and come up with a new hypothesis, redefine your problem statement, and look for new solutions in different areas. You can always be learning, refining, and optimizing. Don’t fret about it!