Is it your job to listen to and analyze social-media messages and direct customer feedback about your product, brand, or service? Are you leading corporate initiatives that leverage consumer-generated content to uncover meaningful insights about your business?
Effective listening and insights analysis allow you to track not just the volume but also the meaning of online conversations across a complex web of consumer-interaction channels.
One of the most common approaches to making sense of customer conversations is to track the presence of keywords within a defined universe of brand conversations (e.g., forums, communities, and social networks).
Although that method highlights text where individual keywords are being used, it is unable to automatically evaluate comments for topics, themes, or sentiment.
Nevertheless, many companies have implemented keyword systems as a first step to help sift through large volumes of consumer-generated content on the social Web. Those companies have then typically hired people to manually read, categorize, summarize, and report on keyword search results at considerable expense—and with questionable accuracy.
Keyword-Based Online Customer Listening: Fraught With Bias and Subjectivity
Selecting and tracking individual keywords in a sea of unstructured content is a process fraught with bias and subjectivity.
Through a combination of personal experience and research, a business user can assemble groups of words that—if contained in a text record—are indicative of a category or sentiment (i.e., "meaning"). Let's say a retention analyst tracks customer attrition by setting up searches for the words or combinations of the words "cancel," "leaving," and "switching." That approach will likely return false positives and miss many records that do not contain the specified combinations.
Moreover, that approach does not provide any information regarding why customers are dissatisfied or at risk. To obtain those insights, the analyst must manually read all the messages flagged.
The way people indicate attrition risk may include hundreds if not thousands of word and phrase combinations. Furthermore, just because a message contains the word "cancel" in the text does not necessarily indicate an attrition risk.
For example, someone could say, "I had to cancel my credit card because I lost it. Could you please call me so I can change my billing information?" Such a message would be included in the example above, even though the customer does not want to cancel service.
In terms of sentiment analysis, keyword-based solutions similarly perform tagging functions based on libraries of positive and negative words. Those "one size fits all" libraries offer the lowest level of accuracy because there are—once again—too many variations on how a customer can express positive or negative emotions in a post.
The value of keyword-based monitoring systems is akin to Web search or press clippings for the Internet—i.e., the tracking of brand mentions across a generic, predefined universe of media sites to gain a general sense of volume and buzz across the widest possible spectrum.
That information can be helpful for high-level corporate metrics and general public-relations tracking. However, the effort required to read flagged content—and analyze it for topic and sentiment—is significant, and it is often accompanied by human subjectivity and fatigue that contributes to the degradation of the resulting insights.
People rarely read, interpret, and categorize content in the same way, and they are notoriously inconsistent and not motivated to deliver insights from such a monotonous process. Accordingly, most attempts to leverage keyword monitoring solutions for detailed analysis are flawed.
Natural Language Processing: A More Accurate Approach to Listening
If keyword-based monitoring systems don't sound like a fit for you, consider customer listening solution based on natural language processing (NLP).
NLP is a scientific approach that enables software to discover and match author intent to a nearly limitless set of words and phrases. NLP-based listening systems assign weights to words and phrases, as opposed to simple "exact match" keyword logic.
In the attrition-risk example, the word "cancel" will most likely produce a strong weight and be a good indicator of an attrition risk; however, the other words that appear with it would also be assigned a weight, all of which would be used to score a verbatim record as true or false for a thematic/sentiment category.
NLP-based processing also relies on machine learning, whereby the system is given examples of text that are "true" for every category of interest. The system then "bubbles up" words that a user would have never thought to enter as search words but that are present in the examples with statistical frequency.
That approach helps eliminate bias, produces a much smaller percent of false positives, and also increases the recall (the number of true messages for a category).
You're probably wondering how a machine can automatically create the expansive weighted word lists and associations that correspond to individual categories.
When given a small sample (a few hundred), a human will categorize the messages with a higher degree of accuracy than any machine. However, when you give a human 100,000 messages to categorize, the human will suffer from fatigue, thus greatly reducing the accuracy.
New customer-listening solutions have been built to leverage a small sample of human categorization but then take that sample and use machine learning to categorize the rest.
That hybrid approach of using humans to train a machine produces categories that are much more accurate (minimizing false positives) and that find more mentions (maximizing recall) than simple keyword searches.
NLP-based approaches to text classification enable business users to create categories that center on a theme rather than a set of keywords. Categories such as Sentiment, Loyalty, and Attrition Risk are not reduced to a collection of representative words but to an overall set of weightings and statistically significant word cues that suggest a topic area.
NLP-based systems perform better than keyword-based systems. Their superiority is considerable, and it doesn't apply to just a few categories;it applies to nearly every category.
Keywords do, of course, have their place. For example, tracking categories that don't contain a theme (e.g., competitor names, product names, etc.) are best handled by simple keywords. But to accurately categorize the theme of a message to help in trending and root-cause analysis, an NLP-based solution is the way to go.
