Focus Group Testing Ban (and Some Better Alternatives)

Focus group testing is appealing. I really do get it. In theory, it’s a quick and efficient way to find out what you need to know about what people think or how they use your product or service, but I’m afraid my experience has confirmed what Malcolm Gladwell says about focus groups: “There are consequences to asking people how they feel or what they like. The consequence is, you’ll be led to a false conclusion, because making people think about their preferences makes them stupid.”

Focus group testing isn’t quick or efficient. And it has a dangerous inclination to lead you down false pathways. This photograph of my daughter was the closest I could get to a metaphor for focus groups: they can lead you into the bushes, and when people start thinking too much about what they think they want—especially in the presence of others—they really can become unsure of whether they should be wearing sandals or roller blades.

Problems with focus group testing

Cutting to the chase, here is a quick account of the problems we have experienced with focus group testing (there are probably more):

1. Expense

Focus group testing is expensive, relative to other better qualitative discovery methods (see below).

2. Bias

Focus group testing is highly susceptible to bias and suggestion.

You can try and ensure that the questions aren’t leading ones (though it’s hard), but you can never truly eliminate the suggestions and influence of the other participants in a focus group.

3. Focus group testing needs focus

The questions you think you want answers to are often not the questions that really need asking. The real questions are very hard to obtain, and are usually not clearly understood at the outset, so if focus group testing is done early its findings are often superficial.

4. Artificiality

The format of a focus group is not analogous to the actual decision-making process, which is more personal and often not shared, and certainly not explained by the decision maker.

5. Moderator and peer influence

We are intensely social creatures, and our ability to detect and respond to the desires of a moderator or a conversation partner are actually quite remarkable. This same ability makes us quite unreliable if someone is trying to understand likely real-world behaviors based on findings from the particular unreal-world social environment created by a focus group.

Here is an absolutely hilarious illustration (thanks to Jimmy Kimmel) of how willing people are to say more or less anything in a public social interaction. It matters not whether they actually have any first-hand knowledge of the subject. It does clearly matter, though, what they think the questioner wants to hear, and neither their true feelings nor their actual likely behavior are illuminated.

6. Stupidity

As Malcolm Gladwell has pointed out, people become stupid when they have to explain why they do things.

The majority of decisions are made emotionally and intuitively; rationalization and the articulation of reasons through speech is secondary (please read Brain Science and Marketing for more on this topic). When people are asked to rationalize why they believe something or behave the way they do, they are actually much more likely to make a different selection than they would have if left to their own devices.

7. Who are the subjects?

People willing to sit through a focus group are a self-selecting group. There are a lot of problems built into this that are difficult to unravel. People willing to do a focus group are a special category of people and are likely not representative. Also, if your goal is to truly understand something, the “non-believers” are just as important as the brand evangelists (and their friends).

8. Shallowness

What truly matters are the deep emotional and even instinctual reasons that a decision is made. The focus group format is not conducive to getting at these critical insights.

What truly matters are the deep emotional and even instinctual reasons that a decision is made. The focus group format is not conducive to getting at these critical insights.

Relevant to both stupidity and shallowness above, I am reminded of the University of Virginia study “Introspecting About Reasons Can Reduce Post Choice Satisfaction” experiment, conducted by Dr. Timothy D. Wilson and colleagues. This experiment used fine art posters and humorous posters (think kittens) to see if people would make different choices if asked to explain their reasons. The subjects chose the art posters 95% of the time if they did not have to explain why, but this number fell to 64% when they had to give reasons. And those who chose kitten posters were later unhappy with their decision. Asking people to think about their decisions, therefore, creates an artificial mental process that is misleading almost by its very nature. If the poster experiment were a focus group, the researchers would have had to conclude that more than a third (36%) of the audience liked kittens when in fact it was really only 5%. The reality would have been that they only liked kittens DURING the focus group.

Here is how Professor Wilson and company summarize their findings: “The present findings add to the mounting body of evidence that thinking about the reasons for one’s preferences can alter decision making in nonoptimal ways.” This academic understatement reads to me like a bullet to the heart of focus group testing.

So what to do instead?

Our preferred discovery and research methods include the following:

Persona Modeling

This discovery method, as outlined in Customer Personas (What Sally Can Show You), is a long interview that we use to ascertain the underlying motivations that drive consumer behavior as it relates to a brand or product. In persona interviews we have found that if the interview subjects attempt to model someone other than themselves—creating a “persona” which stands in for a certain segment of the target audience—the interview goes deeper and can uncover more fundamental motivations than if the interview subjects are simply discussing their own opinions. In effect, they are able to engage in deeper introspection because the fictionalization of the interview has the effect of removing some barriers to revelation. We find that this leads us to a truer and more psychologically profound depiction of the persona’s fears, needs and desires as they relate to a product or service. Our experience with this process has shown that even a slight distance from themselves allows a more candid conversation to take place, and facilitates deeper insights as we dig into what really matters to the consumer.

Behavioral Observation

We watch people interact with the product or service on their own terms in an undirected setting—no guidance, just observation. It is important to understand existing behavior. We are not in the business of advocating that our clients try to change people’s behavior, but rather we help them to understand how their products and services can be made to better interact with existing consumer behaviors.

Long Conversational Interviews

This is not an impersonal interview, but a series of questions that mix the factual with open-ended and personal questions. It requires a skilled and personable interviewer, and facilitates an intimate and honest conversation. Typically the interview subject is very happy about the interview despite its length, and very often new insights can be extracted from such conversations.

Surveys

We also devise and conduct surveys in order to pull in larger data sets. These are useful for both qualitative and quantitative questions that are difficult to extract from the smaller samples above.

Case Study (for those who might be considering focus testing right now)

Here is one real life example that demonstrates with considerable clarity the relative effectiveness of one of our suggested tools, Persona Modeling, compared to focus group testing. In this case the focus group testing was conducted by a highly respected professional research firm. The Persona Modeling was derived from our own custom two-hour persona interviews. We were asked to integrate the findings from the focus groups into our findings for the same target audiences.

The comparison is stark. Above is one of our Competitive Advantage Diagrams, which illustrates the needs of a particular segment of a client’s target audience (represented by a candidate persona) and demonstrates how and if those needs are being met by our client, and whether or not they are uniquely met by the client or by competitors as well. The diagram shows all the specific factors mapped (we’ve left out the factor details in this illustration). Our persona interview factors are shown as numbers, and all the focus group testing factors are shown as letters. We usually do a number of persona model interviews as part of this process, and the relative difference in the findings proved to be about the same in every case. To summarize, the focus groups yielded about 1/3 the information despite costing significantly more.

But that is not all—of critical importance is that the focus group testing also generated false positives, giving the strong impression that certain candidate personas were genuinely viable marketing targets when, in fact, they were shown by our findings not to be. Marketing action based on focus group testing alone would have likely been a very costly mistake.

Finally, the focus group testing was not able to give clear direction regarding which other groups were important for the marketing effort. So focus groups, in our experience, tend not to give clarity where it is most needed, and can spit out false positives. This is very damning. While our discovery makes clear the questions that need to be answered, and allows for carefully focused research to proceed, focus group testing is more likely to burn up money and muddy the water.

‘Nuff said.

Photo by the author