Brexit and Donald Trump’s election were, according to fivethirtyeight.com, well within the margin of error in polling predictions and so were shocking only because people were mentally and emotionally incapable of thinking these events would take place. This made me think of some experiences with focus groups and direct mail back in the days when both were a bigger thing than they are now.
When shown a number of creative options, focus groups would inevitably veer away from the more promotional formats, especially when those formats had big screaming headlines and prominent offers. Yet every time the formats were tested head-to-head in the mail, the promotional format won.
My point: there is no substitute for real-world, boots-on-the-ground testing so long as you have two or more worthy options to consider. People will lie when they think the wrong answer might embarrass them, but not when they’re alone with the offer and their credit card.
Search marketers know this and so do online marketers who constantly refine their landing pages through multivariate testing. But I see many traditional marketers who don’t bother to test or—maybe worse—set up an a/b split and then fail to capture the results or are too busy to analyze them.
I always like to present a more and a less conservative option for any campaign. Of course, I am disappointed when the client chooses the conservative approach and even more disappointed if they test both and the conservative one wins. But the marketplace doesn’t lie. If you ignore this truth, you better be prepared to live with the consequences as well as explain them to your boss or client.