Just like Santa plans to deliver the perfect gift to every child, delivering the right message to the right audience in the ‘most effective form’ is the aspiration of every marketer. The email marketer is in a continuous pursuit of determining the most effective ‘form’ of a message that can elicit a positive action from the target audience.
A/B testing helps the Marketer to compare two or more variants or ‘forms’ of the same message and determine the one that performs better in terms of the desired objective, which could be email ‘opens’ or ‘clicks’. The success of this exercise largely rests on the assumption that the test segments do uniformly and correctly represent the larger target population. Two key factors that need attention are:
- The correct size of test audience that is statistically significant.
- The correct choice of audience to form the individual test segments.
But very often the technique of selecting the audience to constitute the test Segments A and B, is not well defined and is simply based on randomly choosing x% of audience for A/B test. Random selection does not guarantee ‘right selection’ and can often lead to skewed segments, and hence generate results that are not truly representative of the behavior of larger target population. Random selection may not be a big concern for campaigns that are highly focused towards a very well defined target population, but it certainly could produce incorrect predictions for a large size ‘spray and pray’ type of campaign.
Consider an example, where the target population has a demographic mix of Millennials and Gen X. It is important to not just discover this split but to also understand the ratio of this split, and ensure that the test segments reflect this split in the right proportion. Hoping that the business team has a clear idea of the dominant data element could be limiting, especially with growing amount of demographic, and behavioral data being captured daily.
Often determining the most dominant variable in the entire population needs statistical analysis, as the data element may not always be categorical in nature but could also be continuous. Further, the target population is likely to have more than one dominant data element. Examples of these could be demographic elements like gender, age, life cycle, family size, interests, and engagement driven elements like purchase history, email response history, membership history etc. It may be unfair to have one test segment being dominated by audience that is highly digitally engaged with your brand as compared to the other segment, even if both segments uniformly represent a similar demographic profile. It is likely that the response to the A/B Test would significantly vary.
Advanced sampling techniques could be chosen depending on the available data. For example, Proportionate Stratified sampling technique helps create test segments that reflect the proportionate weightage of the dominant data elements as in the entire target population. This means that each segment has the same sampling fraction.
Selection of a statistical technique should be less of a marketer’s worry and should be supported by the tool in an easy to use and visual manner. The key aim should be to leverage available data, and improve the selection of test segments.
Pushkraj MaratheProduct Manager – SAS Institute Inc.