Know-How Exchange : Research/Metrics

Up front confession - I'm a creative director type. But here are my thoughts.

Testing executions after the fact has one big problem: the situation is seldom parallel to the consumer's real media experience. I've had good concepts tossed because focus groups....shown an "animatic" approximation of the spot...dilsliked it because "I don't like cartoons."

Qualitative measures, in particular, can't be trusted for statistically valid results. It's just a few opinions.

I think qualitative methods are helpful in the developmental stage, to help creatives really get to know the prospect and the way he or she thinks. Although, again, not scientifically reliable, they can also provide some insight as to how prospects might respond to a broad creative concept.

I experienced qualitative interviews and biometric responses, both of which provide insights when well-executed, while leading to relatively useless mumbo-jumbo otherwise.

They work best when the creative is at the extreme of being horrible versus brilliant: eg when the first 10 people in your target market tell/show you that the creative sucks, it does not make much sense to go on.

For creatives in the middle of that scale, large sample sizes are often needed to predict which alternative is going to be best received

Results would depend upon what’s being tested (is it a product or service, or a marcom idea?), how it’s being tested, the stage of development you’re in, who it’s being tested with (were reviewers properly screened to ensure they were in the target audience and met category/brand usage criteria?), where you’re testing, and the quality/capabilities of the recruiters and moderator, as applicable.

- I’ve done everything the same and had focus groups in one town where everyone loved the concepts and the next day in another city everyone hated it (make sure you get some geographic dispersion with any testing…)

- On the much participant end, much of it depends upon the due diligence up front and honesty of the research group you’re using – I’ve had researchers tell me I don’t fit their target profile, then use me anyway just to get another ‘body’ or response to fill a quota!

- Ensure you get participants who are able to articulate their responses

- Don’t forget there are significant differences between online qualitative screening and in-person or in-mail responses (each methodology has its own advantages and disadvantages)

- Leverage the benefits of qualitative research: Make sure you’re ‘digging below the surface’ of responses to find out the causal factors/drivers of response (ask more “why?”s)

- Use surveys to narrow down general concepts, do a lot of A/B testing, and consider professional qualitative screeners (like Ideas to Go) to develop ideas or concepts

- Make sure the research format closely represents the venue in which consumers would be naturally exposed to the product or ads

Good luck
-Steve

Qualitative can be helpful if you are choosing between different creative directions, as long as you are willing to hear what the audience really thinks and why. The biggest mistake people often make in using qualitative research (for creative, concept development, or any topic) is to try to project the results onto the overall or target population. If you remember that qualitative is best used when answering the question "why", you can get good feedback concerning how people arrive at their decisions, the emotional-level considerations they are thinking about when making a purchase decision, and how you can tweak brand messaging, creative, or concept development in order to best meet their needs. A good moderator is one of the key aspects of getting the value out of this type of research, as well as the development of a strong moderator's guide.

Some very good input above.

I've had lots of experience using qualitative research for copy evaluation, and most of it has been all but useless. Because you have "real people" reacting, it looks and feels like you're getting good input, but it's impossible to get a good representative sample, so what you're really getting is a few opinions -- which may or may not be representative or helpful.

As jlevin says, qualitative is best used to come up with hypotheses of why people who like an ad like it, and why those who don't don't. Or to find out what words they use to describe their own feelings about a product after seeing a commercial.

But I would NEVER use qualitative research to make a go/no-go decision about advertising creative, whether it's at the concept stage or after the commercial is finished. If a dozen people all think a commercial sucks, that doesn't mean it's bad. It could mean you found the wrong 12 people, or one person influenced all the others, or you're in the wrong city.

Similarly, if they all like it, that doesn't mean it's a winner. If the decision is really important, skip the qualitative research and go directly to a quantitative study.

If what you're looking for is input to the creative process, that's a different story. Qualitative research with a good moderator can uncover some great underlying emotional drivers of behavior, language that people can relate to, segmentation alternatives, and other benefits.

Just don't think you're learning "real truth" from qualitative research.

(I've made this speech to clients dozens of times. They often ignore it until after it's too late. Then they come back and say, "You were right." Saves me from having to make the "I told you so ..." speech.)

Dear Juanisaza,

Qualitative methodologies?

Do you mean: "Who is best qualified or what is the best qualifier of the effectivwness of creative work"?,

Well, before we get into that, what are you testing? To whom are you showing it? Is the test subject the head honcho, the person that calls the shots and who says yeah or nay?

Is the test subject an eventual or potential customer, client, or a competitor?

Is the tester some dude off the street who was lured in with the promise of a tepid diet soda, a slightly soggy hamburger, and their cab fare home?

Or is this person an MBA, "marketing type" gatekeeper, someone in a suit and a nice office who wants a nicer suit and a bigger office? Someone with a reputation they need to massage or create?

The best test for creative work has three parts:

First, was there someone in the company that the work was for with enough guts, authority, commitment, and intelligence—someone who could understand the reasoning they were being given for the work being as it was presetned—for them to take what might have been a leap of faith for them to say "Yes! Let's run it!"?

Second, when it ran, was the marketing material's mix of message, urgency, copy, presentation, establishment of trust, authority, benefit, solution to problem, value, emotional fulfillment, guarantee, social proof, believability, and call to action sufficiently strong enough to move people to opt in, sign up, enquire, or buy? (Notice this list excludes price).

Third, when the sales came in and their numbers were judged against any established benchmarks, goals, projections, competitors, and perceptions of success or failure, when these numbers came in, were the gross and net revenue lines up, down, level, or out of the park in comparison to every variable?

If the answer to the first question is yes, then the second question comes into play and that answer will, as a general rule, answer the third question. It sounds a little like "Britian's Got Talent", but did the contestant get three YES votes?

If it did, it's through the next round.

The toughest question here is the first one, and that answer will always, ALWAYS depend on the the ability of the person presenting the work to weave a tale, to engross his or her audience, and to solve the prospect's problem.

The person in this role needs just the right amount of skill, people knowledge, showmanship, and lobbying ability. There are some who are born to do this kind of work, some who revel in it, For others, it comes with time, often, lots of time.

But do pull this off, the creative work must present a solution to the problem the marketing is addressing and this lies in gathering enough information and coming up with the right direction from which to attack the problem so that it surrenders, or so that the prospect sees the problem vanish as a result of using the good, product, or service in question.

There's a saying in the art world that goes like this:

"I don't know much about art, but I know what I like."

Then there's this little chestnut, roasting on its open fire of creative tripwires and creative direction punji sticks:

"I don't know what I'm looking for, but I'll know it when I see it!"

Here are a couple of questions for you. If whoever it is testing the work, giving their opinion does not "know" about art, or in this case, advertising, how can they comprehend what it is they're looking at and will they judge it against experience? Or will they judge it against their ability to see how swiftly they can kiss what they believe to be the right (or most influential) rear end around to make themselves look good?

If someone does not "know" what they are looking for, what qualifies them to think they will know that they'll "know" what it is that they're looking for when they see it, or if they'd actually recognize it?

How did anyone in either of these groups get into a position of authority that permits them to make the decisions that they are not, in truth qualified to make?

These are questions that you MUST ask and that you must ANSWER before you entrust ANYONE with the authority to judge that which they may not, in truth be qualified to judge.

Remember, how is it possible for anyone in any of these roles to know "it" (whatever the hell "it" is), when the see it if they have no understanding of what "it" is or how "it" works and how long it might take "it" to have an effect at creating a relationship?

Short answer here is, did the advertising increase sales? If it did, the ads were successful. If not, then there was something wrong with the briefing, the message, the offer (if there was one, which there seldom isn't), or the way the problem the marketing was designed to solve was presented.

And all of this—ALL OF IT—has its roots in solid relationships.

A bad ad pitched to an audience that the ad producer already has a relationship with will, as a general rule out pull an ad that might be technically and executionally flawless, but that has zero relationship with the potential buyer.

So yes children, in Adland, ugly ads can out perform beautiful ads.

I hope this helps. Good luck to you.

Gary Bloomer
Wilmington, DE, USA

P.S. Now, here's a call to action for you.

If you've enjoyed reading this entry, click on my name at the top of this entry, visit my profile page, click on the link to my modest blog, and sign up for my free weekly report (I'm writing it now and it'll be available soon). And follow me on Twitter. You know it makes sense:).

Thanks for reading.

Gary B.