The Future of Sentiment Analysis

As Jason has reported before here, sentiment analysis is a tricky thing. Even humans disagree on sentiment 15 percent of the time, so how can a computer create something more accurate? As technology evolves, sentiment analysis gets better, or so we’d like to think.

I caught up with Seth Grimes recently. He is an analytics strategist with Washington, D.C.-based Alta Plana Corporation and a contributing editor at TechWeb’s InformationWeek. He is also perhaps the leading industry analyst covering text analytics. Seth consults, writes, and speaks on business intelligence, data management and analysis systems, text mining, visualization, and related topics. With such an expert on the subject with my reach, I asked him the following:

In layman’s terms, how would you describe/define sentiment analysis?

Sentiment analysis is a set of methods, typically (but not always) implemented in computer software, that detect, measure, report, and exploit attitudes, opinions, and emotions in online, social, and enterprise information sources. (As an aside, what makes it “analysis” is that you’re doing it systematically, with some goal in mind.)

I’ll add is that sentiment analysis much more than simplistically subtracting the number of “negative” words from the number of “positive” in a document or message in order to produce a score.

What does sentiment data and analysis mean for marketers and brands? Is it only about analyzing social media conversations or does it have broader applications?

Yup, it can involve social conversations and also direct and indirect feedback (such surveys, contact-center notes, and warranty and insurance claims), online news, presentations, even scientific papers: Any information source that captures subjective information.

Sentiment analysis lets marketers (and market researchers, customer service and support staff, product managers, etc.) get at root causes, at explanations of behaviors that are captured in transaction and tracking records. Sentiment analysis means better targeted marketing, faster detection of opportunties and threats, brand-reputation protection, and the ultimate aim, profit.

What role do companies like Facebook, LinkedIn, Twitter and Google play?

Interesting choice of platforms. Facebook and Twitter are major sources of sentiment (and also of complementary social connectedness data). Facebook and Twitter accounts have profile data attached to them, but nothing that matches the detailed, usably-structured information you can find on LinkedIn. Google is the ultimate information-access engine, capable of bringing together information from a huge variety of disparate sources, including sentiment information such as product, restaurant, and hotel ratings, although when corporations wish to find, mine, and exploit sentiment they need to turn to deeper BI and analytics tools.

There’s no one-size-fits-all sentiment solution, not Google or one of the several as-a-service solutions out there or any of the capable analysis workbenches or social-media analytics tools. Instead, there’s a whole spectrum of sentiment sources and analysis possibilities.

What are some of the most exciting break throughs that you are seeing in the technology or methodologies related to sentiment analysis?

Wow. First off, there are beyond-polarity solutions, which look at emotional categories — for instance, angry, happy, sad, frustrated, satisfied — that offer much greater business insight and usability than positive/negative/neutral scoring systems. And leading edge solutions are going beyond text, to detect sentiment in speech and even in images and video. On the methodological front, some of the best systems are linking sentiment with transactional records (sales, inquiries, payments, Web clickstreams), including with location correlation, to move us toward a world of integrated analytics.

Can you share an interesting example of a brand or organization has successfully used sentiment?

Let me point you to an article I wrote in February, 2008, where I profiled the use of sentiment analysis (by my friend Tom Anderson of Anderson Analytics) as one analytical component of “triangulation” strategy around the Unilever Dove-brand pro.age campaign. I quote Catherine Cardoso of Unilever in the article: “We were very pleased with the results and the depth of insight. The results were helpful beyond understanding reactions to our campaign. We also gained an understanding of what motivates people on discussion boards, which issues are most important to women in our target group, and how to create better products and messaging for them.”

In your opinion, what is the biggest obstacle keeping sentiment analysis from reaching its true potential?

Misperceptions, also inflated expectations, fostered by low-grade tools that are keyword based and lack any mechanism to link sentiment to actual business outcomes. On the one hand you get low accuracy, and further there’s a “decision gap.” You get a colorful dashboards, but because the tools are working in isolation, treating social and survey sources as information silos, you can’t reliably know what sentiment is important, and what sentiment really means to your business in the sense of driving transactions, boosting satifaction, and so on.

Companies like Radian6, (recently bought by Salesforce), SM2 and others have included sentiment as part of their social media analytics/monitoring tools for some time now. How will the future of the technology and method to using the data be different than what is available now?

Let’s name names. I was incredibly disappointed when I get an SM2 briefing from Alterian. I saw, earlier this year, primitive sentiment capabilities and analysis interfaces that resembled those of BI tools circa 2000. Check out what I wrote last March in an article, “What I Look For In A Social Analysis Tool.” There are other tools with similar, serious deficiencies.

Radian6, as an example, illustrates two routes forward. First, Radian6 provides a framework for plug-in of dozens of disparate extensions, including text and sentiment analysis from at least four different providers, AlchemyAPI, Clarabridge, Lexalytics, and OpenAmplify. This sort of openness and inter-operation is a benefits the solution providers and their customers alike. Second, the acquisition of Radian6 by Salesforce.com should further users’ ability to link customer-relationship data captured in Salesforce — profiles, transactions, interactions — with customer attitudes, emotions, and opinions, posted on-social and online and analyzed via Radian6, related to products and services offered by Salesforce customers and their competitors. This would mark the end of social as a silo.

What impact will mobile technology and the context that it provides (IE: location specific data) have on collecting and analyzing consumer sentiment?

Mobile creates the opportunity to solicit and collect feedback on the spot, at the point-of-service, and to understand peoples’ choices as they do whatever they’re doing. (Of course you have to consider privacy regulations and expectations.) And by collecting location and time data along with sentiment, linked to activities, you get additional analysis variables that can feed more capable, more accurate predictive models. Mobile is huge, in many, many ways.

You are one of the organizers of the upcoming Sentiment Analysis Symposium. Why do you feel an event like this is needed now? What are you hoping to accomplish?

The up-coming symposium, November 9 in San Francisco, is actually the third, following on April 2010 and 2011 New York events. The events are designed as a meeting ground for technologists and business users. They’re designed for education, for new and experienced folks alike, and to create networking and deal-making opportunities.

Folks should check out the Web site, sentimentsymposium.com, also the optional, pre-symposium tutorial (for people just getting started) and research session (for advanced users and developers).

How can people find out more information about how sentiment analysis is being used for business?

I’ve written quite a bit on the topic myself, and you can also learn from solution providers. Ask them for case studies and customer references. But I also recommend just taking a shot yourself. There are two factors that support this approach —

1) You’re working with natural language, with material you can understand directly, and it’ll be pretty clear whether the tools you’re trialing are doing a good job.

2) You have a variety of choices available including hosted, as-a-service solutions that can be used without a large up-front investment, also social-analytics and survey-analysis solutions that embed sentiment analysis without imposing a heavy technical burden on users.

I’m happy to field questions. Folks should contact me via Twitter at @sethgrimes at 301-270-0795 or grimes(at)altaplana.com.

—————

Use the code FOAF for a $100 discount your registration to attend the Sentiment Analysis Symposium on November 9th.