In our explorations of social media monitoring firms, one differentiating factor between services is often the ability to parse out sentiment and tone of online conversations found about your brand. This task is performed by sophisticated computer functionality called natural language processing (NLP). It’s not the sexiest topic in the world, but it’s a very important one for public relations professionals, marketers and brand managers to understand.

Fortunately, I caught up with Jeffrey Catlin, the CEO of Lexalytics, an Amherst, Massachusetts-based NLP firm, recently to dive into the subject matter a bit. In this episode of SME-TV we learn what drives natural language processing, what the types of accuracy rates should be expected when using tools that provide the functionality and that Lexalytics has some consumer-facing products on the horizon we can be on the lookout for.

Exploring Natural Language Processing With Lexalytics CEO Jeff Catlin from Jason Falls on Vimeo.

As a follow-up to the chat with Catlin, I spoke with a couple of my friends in the monitoring space. A couple of them agreed with most of Catlin’s assessments of the accuracy question. One, however, noted that their service does use a vendor like Lexalytics to provide a foundation of NLP for their tool, but has built an ever-learning algorithm atop that technology. Because they can train the tool to understand the dynamics of conversations around a particular client and build upon that learning in an ongoing fashion, they can produce accuracies in the low- to mid-80s over time and are even around 83% on some clients now.

Another NLP hang up is that tools like Lexalytics were trained to interpret press releases and more formal journalistic pieces. Social media is chock full-o-typos and other truly natural human language quirks that the machines can learn, but only if they’re told to do so. Plus, there are dozens of sentiments around brands, not just positive, negative and neutral, so NLP has its limitations. Still, as Catlin said, with a large amount of information, it can give you a broad brush stroke of look at what’s out there.

Regardless of the tit-for-tat debate over who is more accurate or if they even are, at least we know understand the process and environment better. Hopefully, this helps you make better decisions in social media monitoring solutions for your company or brand.

Be sure to keep an eye on Lexalytics in the coming weeks, however. Their free tools, which I’ve seen demonstrations of, are simple, intuitive and most useful for those of us who have data (perhaps blog content, customer surveys or focus group interview transcripts) and need a tool to process and analyze it without chewing up a lot of budget. I’m excited to see how you can leverage those tools.

And, if you have more questions about natural language processing and its application in the social media monitoring and measurement space, please throw them out in the comments. If I can’t answer them, I’ll do my very best to find someone who can. (Plus, I’ll bet the Lexalytics and social media monitoring folks will be reading with interest.)

Reblog this post [with Zemanta]
Did you enjoy this blog post? If so, then why not:Leave Comment Below | Subscribe To This Blog | Sign Up For Our Newsletter |

About Jason Falls

Jason Falls

Jason Falls is the founder and chief instigator for Social Media Explorer's blog and signature Explore events. He is a leading thinker, speaker and strategist in the world of digital marketing and is co-author of two books, No Bullshit Social Media: The All-Business, No-Hype Guide To Social Media Marketing and The Rebel's Guide To Email Marketing. By day, he leads digital strategy for CafePress, one of the world's largest online retailers. His opinions are his, not necessarily theirs. Follow him on Twitter (@JasonFalls).

Other posts by

Comments & Reactions

Comments Policy

Comments on Social Media Explorer are open to anyone. However, I will remove any comment that is disrespectful and not in the spirit of intelligent discourse. You are welcome to leave links to content relevant to the conversation, but I reserve the right to remove it if I don't see the relevancy. Be nice, have fun. Fair?

  • http://twitter.com/jeffhora Jeff Hora

    Thanks for the interview. I'll really be interested on how the consumer service and plug-in take off, since that is a way to evangelize capabilities and value (especially since so many may not get what it does/can do).

    • vandolet

      Wow Jeff, thanks for the awesome, very clear explanation.
      I will be quoting some of your phrases (particularly, “Humans are only 85-90%”) when discussing sentiment with our clients. Great post Jason!

      Michael Fraietta
      Filtrbox
      @Michael Fraietta

      • http://socialmediaexplorer.com JasonFalls

        Thanks Michael. Glad to see you here.

  • http://twitter.com/jeffhora Jeff Hora

    Thanks for the interview. I'll really be interested on how the consumer service and plug-in take off, since that is a way to evangelize capabilities and value (especially since so many may not get what it does/can do).

  • Michael Fraietta

    Wow Jeff, thanks for the awesome, very clear explanation.
    I will be quoting some of your phrases (particularly, “Humans are only 85-90%”) when discussing sentiment with our clients. Great post Jason!

    Michael Fraietta
    Filtrbox
    @Michael Fraietta
    http://fbx.bz/E

  • nilesh_bansal

    Hi Jason,
    I agree with you. Language processing is a very important component when you go beyond the world of social media monitoring to the space of analytics. There are two main parts to it:

    – Summarizing the text to automatically create condensed visual summary of what is going on, without having to read through the data.

    – Finding the sentiment. While it is a difficult problem, based on our work here at Sysomos, it is doable. The trick is to make the engine understand the context not only within well written news articles but also in more colloquial social media language.

    NLP however goes beyond just sentiment or summarization. We use it in all modules of the Sysomos platform, such as in cleaning spam and removing junk.

    Nilesh Bansal
    http://www.sysomos.com

    • http://socialmediaexplorer.com JasonFalls

      Thanks for chiming in Nilesh. Glad to have your perspective here.

      • nilesh_bansal

        Jason, not sure if you have had a chance to look at Sysomos MAP. If not, we would love to show you a demo.

        • http://socialmediaexplorer.com JasonFalls

          It's a deal. Give me a week or so and we'll take a look under the hood.

  • nilesh_bansal

    Hi Jason,
    I agree with you. Language processing is a very important component when you go beyond the world of social media monitoring to the space of analytics. There are two main parts to it:

    – Summarizing the text to automatically create condensed visual summary of what is going on, without having to read through the data.

    – Finding the sentiment. While it is a difficult problem, based on our work here at Sysomos, it is doable. The trick is to make the engine understand the context not only within well written news articles but also in more colloquial social media language.

    NLP however goes beyond just sentiment or summarization. We use it in all modules of the Sysomos platform, such as in cleaning spam and removing junk.

    Nilesh Bansal
    http://www.sysomos.com

  • researchgoddess

    Jason, thanks for posting this! Semantic search is becoming a hot topic in the recruitment sourcing world as well; this was a very good explanation of how to determine sentiment and I am surprised at even the human error here. Great interview!

    • http://socialmediaexplorer.com JasonFalls

      You're welcome, Amy. Thanks for the comment.

  • researchgoddess

    Jason, thanks for posting this! Semantic search is becoming a hot topic in the recruitment sourcing world as well; this was a very good explanation of how to determine sentiment and I am surprised at even the human error here. Great interview!

  • http://socialmediaexplorer.com JasonFalls

    Thanks Michael. Glad to see you here.

  • http://socialmediaexplorer.com JasonFalls

    Thanks for chiming in Nilesh. Glad to have your perspective here.

  • http://socialmediaexplorer.com JasonFalls

    You're welcome, Amy. Thanks for the comment.

  • nilesh_bansal

    Jason, not sure if you have had a chance to look at Sysomos MAP. If not, we would love to show you a demo.

  • http://socialmediaexplorer.com JasonFalls

    It's a deal. Give me a week or so and we'll take a look under the hood.

  • jeffcatlin

    Jason, Thanks for taking the time with me out in the wind storm at Gillette. Seems I'm already late to the commenting party, but I did want to comment one the person that said it's possible to achieve mid-80s levels of accuracies on sentiment. This is certainly true, if you focus in on a vertical market or particular type of content. We ourselves did a project to tune our engine for Hotel Review content and managed to achieve about 84% accuracy on that data, so yes it's possible but does require some added work.

    Thanks to those that liked the “simplified” terminology I used… Nothing worse than people that explain technology using big nasty words :-)

    • http://socialmediaexplorer.com JasonFalls

      Thanks for the time, then and now, Jeff. Great to hear how NLP works
      from someone who knows. Can't wait to see the new consumer products as
      well.

    • http://twitter.com/MarkAChaves Mark A. Chaves (SAS Institute)

      Jeff is spot on here… especially with regard to application of NLP within vertical industries… we find that, with our customers, that success is driven by developing “rules” within very specific verticals and the context within those verticals (e.g. ongoing “corporate reputation” issues, known consumer issues/needs, etc.).

      Furthermore, by enabling domain experts across customer touch-points (marketing, PR, customer service) to influence how rules get “taught”, accuracy rates can approach the 80% range… any social media measurement / NLP efforts must be seen as a journey…

  • jeffcatlin

    Jason, Thanks for taking the time with me out in the wind storm at Gillette. Seems I'm already late to the commenting party, but I did want to comment one the person that said it's possible to achieve mid-80s levels of accuracies on sentiment. This is certainly true, if you focus in on a vertical market or particular type of content. We ourselves did a project to tune our engine for Hotel Review content and managed to achieve about 84% accuracy on that data, so yes it's possible but does require some added work.

    Thanks to those that liked the “simplified” terminology I used… Nothing worse than people that explain technology using big nasty words :-)

  • http://socialmediaexplorer.com JasonFalls

    Thanks for the time, then and now, Jeff. Great to hear how NLP works
    from someone who knows. Can't wait to see the new consumer products as
    well.

  • http://www.spiral16.com/spark/virtualization-engine/ Eric Melin

    Really great interview. I agree that what makes automated sentiment classification valuable is the ability to uncover trends and make sense of enormous data sets that human vetting just can't accomplish. The rates of accuracy described here ring true; it's still a great overview or “broad brush stroke” tool, as you said.

    It becomes vitally important then to make sure you integrate that data into a usable toolset. A virtualization can give you a visual representation that's easier to gain insight from than simple pie charts and graphs.

  • http://www.spiral16.com/spark/virtualization-engine/ Eric Melin

    Really great interview. I agree that what makes automated sentiment classification valuable is the ability to uncover trends and make sense of enormous data sets that human vetting just can't accomplish. The rates of accuracy described here ring true; it's still a great overview or “broad brush stroke” tool, as you said.
    It becomes vitally important then to make sure you integrate that data into a usable toolset. A virtualization can give you a visual representation that's easier to gain insight from than simple pie charts and graphs.

    Eric Melin – @ Spiral16

  • http://nlpworldblog.blogspot.com/ anwer

    Great video

  • http://twitter.com/dshimy Darian Shimy

    Jason, great article. I couldn't agree more with the Jeff's comments on accuracy of sentiment and it's usefulness in aggregate. We developed a hybrid sentiment engine that handles poor grammar and typos so I understand first-hand the difficulty in handling social media. Sarcasm is a huge challenge. Thanks!

    – Darian Shimy, VP of Technology Biz360.com

    • http://socialmediaexplorer.com JasonFalls

      Thanks, Darian. Great to hear from you!

    • http://www.easyrecovery.ie/ Data Recovery

      Yes, I agree to you that this is really a great article.

  • http://twitter.com/dshimy Darian Shimy

    Jason, great article. I couldn't agree more with the Jeff's comments on accuracy of sentiment and it's usefulness in aggregate. We developed a hybrid sentiment engine that handles poor grammar and typos so I understand first-hand the difficulty in handling social media. Sarcasm is a huge challenge. Thanks!

    – Darian Shimy, VP of Technology Biz360.com

  • http://socialmediaexplorer.com JasonFalls

    Thanks, Darian. Great to hear from you!

  • winter123

    Louis Vuitton Matsuya Ginza Store done renewal and displays their “speedy” archives over past years tiffany jewellery online shop sells varieties of tiffany jewelry products, including tiffany bracelet, tiffany necklace, tiffany ring, tiffany earrings, tiffany pendant and various other tiffany replica jewelries. We are one of the biggest suppliers of tiffany jewellery sale
    ,Louis Vuitton has been welcomed by many louis vitton handbag lovers. I think there is no cheaper louis vuitton handbags than you in the world. “speedy” is known as Louis Vuitton’s iconic line. Every arranged bags were very interesting.

    Louis Vuitton Matsuya Ginza Store done renewal and displays their “speedy” archives over past years,Louis Vuitton has been welcomed by many Louis Vuitton Handbags lovers. I think there is no cheaper LV Handbags than you in the world. “speedy” is known as Louis Vuitton’s iconic line. Every arranged bags were very interesting.
    Louis Vuitton Matsuya Ginza Store done renewal and displays their “speedy” archives over past years,Louis Vuitton has been welcomed by many Louis Vuitton Handbag lovers. I think there is no cheaper louis vuitton sale than you in the world. “speedy” is known as Louis Vuitton’s iconic line. Every arranged bags were very interesting.

  • winter123

    Louis Vuitton Matsuya Ginza Store done renewal and displays their “speedy” archives over past years tiffany jewellery online shop sells varieties of tiffany jewelry products, including tiffany bracelet, tiffany necklace, tiffany ring, tiffany earrings, tiffany pendant and various other tiffany replica jewelries. We are one of the biggest suppliers of tiffany jewellery sale
    ,Louis Vuitton has been welcomed by many louis vitton handbag lovers. I think there is no cheaper louis vuitton handbags than you in the world. “speedy” is known as Louis Vuitton’s iconic line. Every arranged bags were very interesting.

    Louis Vuitton Matsuya Ginza Store done renewal and displays their “speedy” archives over past years,Louis Vuitton has been welcomed by many Louis Vuitton Handbags lovers. I think there is no cheaper LV Handbags than you in the world. “speedy” is known as Louis Vuitton’s iconic line. Every arranged bags were very interesting.
    Louis Vuitton Matsuya Ginza Store done renewal and displays their “speedy” archives over past years,Louis Vuitton has been welcomed by many Louis Vuitton Handbag lovers. I think there is no cheaper louis vuitton sale than you in the world. “speedy” is known as Louis Vuitton’s iconic line. Every arranged bags were very interesting.

  • jtorras

    Thanks for this article Jason!
    At Inbenta we are using our “semantic clustering” technology to group conversations that have a similar topic. As you told, they are not necessarily good or bad, but it's interesting to see patterns related to semantic meaning of those conversations

    • http://socialmediaexplorer.com JasonFalls

      You're welcome. Thanks for the comment!

  • http://www.transferpod.com/features.php transfer ipod to computer

    This is really a great article. Thanks for posting.

  • http://www.transferpod.com/features.php ipod music to computer

    Wow! You have a great point in your topic. Well I came to realize of what you pointing out. Thanks for sharing this great great idea to us.

  • http://www.transferpod.com/features.php ipod music to computer

    You had a great post, but I can't relate in it.

  • Pingback: Why You Shouldn't Trust Automated Sentiment Scoring

  • Pingback: Why You Shouldn’t Trust Automated Sentiment Scoring | BrettMBell.com

  • http://www.fordpartsgateway.co.uk/fordkaparts.php Ford KA Parts

    Because they can train the tool to understand the dynamics of conversations around a particular client and build upon that learning in an ongoing fashion, they can produce accuracies in the low- to mid-80s over time and are even around 83% on some clients now.

  • http://www.fordpartsgateway.co.uk/fordkaparts.php Ford KA Parts

    Because they can train the tool to understand the dynamics of conversations around a particular client and build upon that learning in an ongoing fashion, they can produce accuracies in the low- to mid-80s over time and are even around 83% on some clients now.

  • Tan_yong_tai

    Interesting. Thank you.