My early direct mail copy chiefs beat this mantra into me: only test one thing at a time. If you change the offer and simultaneously change the letter lead, how do you know which change was responsible for any lift?
I thought of this as the drumbeat of “Free FICO Score” offers from Chase Slate continued, and I realized one of the envelopes looked different. Same color scheme, virtually the same OE copy on the front but arranged in a slightly different way. The first really noticeable change is on the back. One says “no annual fee” and the other has a lineup of four unidentifiable awards. (We know which would win in that test, don’t we, since benefits always eclipse chest pounding.)
Inside the “alike but different” motif continues. One letter starts “Transfer high rate balances from other credit card issuers and save money.” The other, “From balance transfers to new purchases, Chase Slate makes saving simple.” Exact same facts, but one is about “you” and the other about “the card” so again, it’s pretty clear which would win if tested on its own. (That old copy chief of mine would have had the second writer start over, rather than testing the two leads.) However, the “you” copywriter is paired with the art director who put the shields on the back of the OE, so we’re tied.
And it continues throughout the package, with design and copy slightly different without changing the facts or the basic presentation. What’s happening here is that two teams were tested against each other to see which one is “better”—a costly experiment on Chase’s part. This isn’t the same as a direct mail package test in which creative teams come up with completely different ideas from scratch. It’s an expensive waste of time.
My advice to Florian Egg-Krings, who signs both letters (no testing variations there): test spelling out “credit score” on the OE rather than calling it “FICO score”. Take one key benefit—I’d probably go with the reasons you’d want a monthly credit report and how great it is to get it for free—and lead one letter with that, then keep the other about your laundry list of benefits. Now you’ve got something worth testing.