'I Was Told There Would Be No Math.'

I am no longer satisfied looking at program dashboards that roll up multiple messages and present average/mean performance numbers without a measure of variance. This wasn’t always the case.

When I first cut my teeth as an email marketer in the late 1990s, it had been a while since I’d taken a math class. Having attended a small liberal arts school in the Midwest, I was never asked to crunch a single number, and happily concentrated on the subjects that most interested me at the time—Cultural Anthropology and Sociology. So when I went to report on those first campaigns, I didn’t have a whole lot of tools in my belt to analyze them. I calculated the basic performance metrics—open rate, click through, etc.—and when trends or anomalies presented themselves I took notice.

Since those good old days, I have brushed up considerably on my analytical skills. Now I still rely on many of the same KPIs, and a few more, to judge program performance, but I definitely see the data in different ways.

As mentioned above, now I can’t look at a set of data without calculating their standard deviation. It’s of course useful to know, for example, what the average open rate is for a group of messages, but without a measure of variance, the average doesn’t yield much useful information. For example, I could have two different campaigns of three messages each that both have an average open rate of 30%. If we stop there, we could deduce that both programs are performing similarly with regards to opens—when one program could be comprised of messages with open rates of {29%, 30%, 31%} and the other could be {3%, 19%, 68%}. The standard deviation of the open rates for the first campaign is 1—not much variance—but the standard deviation of the second is 33.87, indicating some substantial fluctuations. Again, both have mean open rates of 30%, but there are probably different things going on in data set #2 that warrant further exploration.

If you don’t currently calculate the standard deviation of your campaign metrics, I suggest you try it. Don’t concern yourself that the standard deviation is the “root mean square deviation of values from their arithmetic mean,” just type =STDEV into excel and select the data. It only takes a second, and I believe you may find the information useful.

—Nicholas Einstein of Datran Media