$500 million is a big sum by anyone’s measure. Now imagine using email to deliver most of that money in just 18 months. Toby Fallsgraff and his team raised a sum not far off that amount for the Obama campaign last year. In fact, the digital department on the Obama campaign accounted for the bulk of the election funds raised from donors.
The strategy behind this incredible success involved something the email team called “Frankenstein testing”. I was fortunate enough to catch-up with Toby and he shared a few details of his strategy.
Three things that were key to success:
- List size – the campaign did have a big quality database to leverage
- Email frequency – OK, so a lot of email was sent
- Test and optimization – up to 6 times more revenue was produced per campaign by testing
I’m going to share with you their approach to the test and optimization piece of the puzzle. In Toby’s own words “we did as much testing as humanly possible”.
The fundraising emails were all about getting donations and so copy had to do most of the work persuading donors to open their wallets. The process used is as follows:
- A team of 18 writers created 4 to 6 different body copy messages for every campaign.
- Each body copy variation was sent out to test audiences with three different subject line choices.
- Large test cells of 50,000 contacts were used to determine winners quickly.
- The most important evaluation metric was simply the number of donations.
- The test cells were usually sent around 7am and a winner picked about 60 to 90 minutes later.
- The winner went to the remaining database, but some segments received slight variations on the winner depending on their donation history.
You may be wondering how 18 writers could be afforded. Simple, the uplifts seen more than paid for them. On the ‘I will be outspent campaign’ of June 2012 the winning campaign raised $2.5m and a projected $2.1m more than the worst performing cell. That pays for the copy writers and the 10 analytics engineers checking over the results.
Occasionally although there was a winner, none of the emails performed to the level expected and in these cases the whole send was canned. The mentality was not one of send unconditionally.
Businessweek reported the different subject lines tested for the June ‘I will be outspent’ campaign. I quizzed Toby about a missing part of that story. The importance of the message body copy combined with the subject line.
The body copy plays a vital role too; the subject line can’t do all the work by itself.
In some cases a generic subject line was used to allow an evaluation of the body copy alone, such as use of the subject line ‘Hey’. This could be matched and make sense with any other body message.
However, an important learning was that the combination of subject line and body copy was found to be important.
Toby explained these Frankenstein tests. The best bits of messages from different test cells were combined in the hope of creating monstrous results (cough, sorry). A winning subject line was taken based on open rate and then combined with winning body copy message based on donations. Obviously this could only be done when the subject line was not so specific to the body that it would not have sense.
The new combined messages were the Frankenstein’s and these then tested too.
What was interesting was that the Frankenstein’s’ mostly didn’t turn out to be winners. Sometimes yes, but mostly not.
So the combination of a great subject line with great body copy was not necessarily a great email – often it lost in a head to head test with the control segment.
The insight here is that the subject line impact is much more than just getting the open. How the subject line frames the subscribers thoughts and how they perceive the whole message as a result of the subject line plays a big role.
Subject lines must be optimized for the action and not just the open.
You’re surely curious by now about some of the actual subject lines tested. Here’s twelve from the June ‘I will be outspent’ campaign:
- I will be outspent
- Some scary numbers
- If you believe in what we're doing
- Last call: Join Michelle and me
- Would love to meet you
- Do this for Michelle
- The most popular Obama
- Michelle time
- Deadline: Join Michelle and me
- Thankful every day
- The one things the polls got right…
The top one is the best performer, raising $2.5 million and the bottom the worst. Keep in mind that the subject lines had different body copy associated with them, so the subject lines can’t be evaluated on their own. Since as the Frankenstein tests verified, the combination of subject and body message is important.
Finally, I should be clear it wasn’t just subject lines and body copy that was tested, other items tested included; calls to action, donation amounts, text formatting and emphasis.
Acknowledgement: My sincere thanks to Toby for taking the time to share this information with me.