A false god?
Graham Booth
9th March 2010
My heart sank the other week when a client asked me to use a customer segmentation algorithm to recruit respondents for this week’s groups. I’ve been here before: “Thou shalt worship at the altar of the bright shiny customer segmentation model that is now driving all of the company’s business analysis and strategy; for it is good, and thou shalt not cast aspersions thereon, or thou shalt be cast out into the night.”
Sure enough, the algorithm was rubbish. On first glance, it didn’t feel right; on closer examination, it just didn’t work. If you followed the template that was supposed to spit out a respondent in the right segment, you were going to wind up with someone who logic dictated simply didn’t exist or, if they did, was not going to be representative of the type of person supposedly characterising the segment. You didn’t have to be a rocket scientist to work this out, you just had to be able to think logically and apply a little common sense.
What’s more, there was simply a mistake in there, a straightforward coding error that would lead you in exactly the wrong direction. This was the client’s established segmentation study and no-one had yet spotted the error.
On a previous occasion for another major client, I was easily able to prove, using the figures contained within the client’s own algorithm guide, that the percentage of the universe that would be recruited using the algorithm was many times smaller than the percentage of the population the original study stated fitted into the relevant segments. If we could find respondents who did fulfill all the requisite criteria, they’d clearly be freaks! My protestations fell on deaf ears. What is it about quantitative data that makes people disengage brain and blithely accept what a little clarity of thought will clearly demonstrate is tosh? It took over a year and several projects before the Head of Insight finally and reluctantly accepted that the segmentation was completely unsuitable for use in qual recruitment.
Now, while it’s a pain having to explain to clients why their precious segmentation doesn’t work for qual, it’s clearly something we can get around, by understanding the spirit of the segment and designing proxy criteria to ensure we find the right people.
However, what if it’s not just the segmentation algorithm that’s tosh? What if this is just a symptom of a segmentation study that is itself deeply untrustworthy? And since a lot of algorithms don’t work, does this mean there are a lot of questionable segmentation studies out there? I can see several reasons why users may not realise or accept that their segmentation study is flawed: it has the gloss of credibility that numbers bestow and few people have sufficient confidence in their numerical analysis skills to challenge the data; it cost too much to discard; and high level managers have bought into it. It does make you wonder... .
I’ll leave the (nearly) last word to my favourite qual-minded quantie, Albert Einstein: “Not everything that counts can be counted, and not everything that can be counted counts”. But, in the case of segmentation algorithms, you really don’t need to be Einstein to see the problem.Comments
Added at 12:45pm on Tuesday 9th March 2010
An enlighted client will always be willing to refine...or even revise...a segmentation model, whether because the market has changed or because better definitions have been identified.
Recent experience with a financial client led to precisely this - like you, there was a "black hole" in the model that meant that a significant chunk of their indirect distribution channel could not be categorised. Yes, it takes a degree of courage to accept shortcomings, but the cost of not doing so is usually too high to contemplate.
Re a "sub optimal" segmentation model being unsuitable for qual recruitment, the problem extends to recruitment/quotas and analysis of subsequent quant studies as well. When something is rotten, it's often rotten through and through!
Added at 8:42am on Wednesday 10th March 2010
One of the problems that I've encountered with quant segmentations is that users sometimes think that all the people in a particular segment meet all the describing criteria completely, and all of the time. So, instead of interpreting a segment as a cluster of the population who tend towards fitting a number of different criteria (demographics, usership, attitudes etc.), it is seen as a rigid, discrete block of people all fitting all criteria 100%.
Again, common sense says that human beings simply aren't like that, but a long list of segment attributes then gets translated into a rigid and unworkable qualitative recruitment questionnaire. Like Graham, I usually argue for developing a description for recruiters that is in the spirit of the segment - but don't always succeed!
Added at 4:57pm on Wednesday 10th March 2010
Another problem I have encountered is clients who drift from the original segmentation facts and romanticise their segments - they give them charming names, extrapolate their characteristics and attitudes, and invent habits that suit the client's marketing strategy rather than the facts. They then send us out out to recruit them, and are slightly surprised when they don't get a group of cookie-cutter stereotypes and when their respondents disagree with the 'insights' that underline the client's concepts.
So few clients really understand what a segmentation study really gives them, and how very similar their segments often are.
Leaving a comment
If you're an ICG Member, you can comment on this blog entry. Log in to the site, then you'll be offered a form to allow you to enter your comment while viewing each blog...
