Blog, Email Marketing Tools

Differences between a real-time A/B test and a traditional A/B test

29 April 2015 Comments Off on Diferencias entre un A/B test en tiempo real y un A/B test tradicional

Contrary to what might appear at first glance, The email marketing industry is at one of its most exciting times, with the emergence of new vendors and innovative technology solutions. Two companies that are leading this movement and which we at Digital Response follow very closely are 8 Seconds y Movable Ink. These are companies that offer solutions for real-time email marketing and contextual marketing.
One of the most obvious applications for email marketing professionals offered by companies such as these is the creation of Real-time A/B testing. The great advantage of real-time A/B testing is that the "winning" version is shown to all users except those "necessary" to determine that one version is better than another. Let's see it with an example:

  • Traditional A/B testSuppose we want to test two different CTAs and we apply the 10-10-80 method. This method consists of sending to one 10% in the database version A, to another 10% version B and, after sufficient time to have obtained conclusive results, to the remaining 80% the winning version.. In absolute numbers, if the database was 200,000 users, the winning version would reach 180,000 users.
Método 10-10-80
Method 10-10-80
  • Real-time A/B testing. In this method, We send, in a first and unique shipment, version A to the 50% of the base and version B to the remaining 50%. After the first few hours (let's say 2 hours) there will have been a sufficient number of openings of version A and version B to draw statistically conclusive conclusions as to which of the 2 versions is obtaining a higher CTO.. As soon as it is concluded which version is the winner, the "losing" content (the images) of the emails that remain in the mailboxes without having been opened yet, will be replaced by the winning version. When the emails are opened, the good version will be displayed. In the example of the 200,000 user database, we see how a real-time A/B test will show the winning version to 195,000 people, that is 15,000 more users than the traditional version.

Real Time A/B test
Real Time A/B test

As we can see, carrying out an A/B test in real time allows you to optimise the results, thus maximising the reach of the winning version. We are confident that more and more companies will opt to introduce this type of innovation in their email marketing programmes.