What is sentiment analysis and how does it work?

So you want to know more about Natural Language Processing (NLP) sentiment analysis? Well, then you’ve come to the right place. In this article, we’ll help you understand how sentiment analysis works, explain how the Lettria platform has been developed to make this process easier and more powerful, and provide you with some examples of business use cases and applications for sentiment analysis. 

The TL;DR explanation? Sentiment analysis (sometimes referred to as opinion mining or emotional artificial intelligence) is a natural language processing technique that analyzes text and determines whether the data is positive, negative, or neutral.

The applications and use cases are varied and there’s a good chance that you’ve already interacted with some form of sentiment analysis in the past. But before we get into the details on exactly what it is and how it works, let's (all too) quickly cover the basics on natural language processing.

{{ligne}}

Natural Language Processing

Natural language processing allows computers to interpret and understand language through artificial intelligence. Over the past 50 years it has developed into one of the most advanced and common applications for artificial intelligence and forms the backbone of everything from your email spam filters to the chatbots you interact with on websites. 

Through machine learning and algorithms, NLPs are able to analyze, highlight, and extract meaning from text and speech. In its simplest form, this could simply be using an artificial intelligence (AI) model to scan text to identify particular words or phrases, but in its more advanced applications, as we will touch on later, it allows computers to identify the emotion of the speaker themselves — sometimes better than even a human can manage. 

NLPs are typically employed when an organization can benefit from analyzing sets of data that are far too large to process manually. Whether it’s trying to monitor social media or emails or extract information from reviews or exchanges with customers, there’s a good chance that having humans analyze the information will be too time consuming and resource heavy. 

Syntactic analysis (sometimes referred to as parsing or syntax analysis) is the process through which the AI model begins to understand and identify the relationship between words. This allows the AI model to understand the fundamental grammatical structure of the text, but not really the text itself. For example, sentences can be grammatically correct and not make any sense, or it could fail to identify the contextual use of some words as a result of the sentiment or emotion within the text (sarcasm being a common issue).

We won’t go into too much detail about exactly how an NLP works in this article, but they use text vectorization to transform the data into a format that the AI model can understand and then process the textual data using machine learning algorithms that allow the computer to make associations between an input and the correct output. 

This process means that the more data you feed through your NLP the more accurate it becomes. With each new analysis allowing it to build a more complete knowledge bank that helps it to make more accurate and complete analysis.

NLPs have now reached the stage where they can not only perform large-scale analysis and extract insights from unstructured data (syntactic analysis), but also perform these tasks in real-time. With the ability to customize your AI model for your particular business or sector, users are able to tailor their NLP to handle complex, nuanced, and industry-specific language.

You can find out a lot more about the benefits and applications of an NLP in other sections of our website, but a platform like Lettria allows users to go beyond that simple textual analysis and actually identify the sentiment and emotion expressed within the text itself. 

{{ligne}}

‍What is sentiment analysis? 

So we’ve given you a little background on how natural language processing works and what syntactic analysis is, but we know that you’re here to have a better understanding of sentiment analysis and its applications.

AI models in natural language processing can begin to gain a more complete understanding of language by using sentiment analysis to understand both the polarity of text (positive, negative, or neutral) and the specific emotions expressed within a text. 

Sentiment analysis is the foundation of many of the ways in which we commonly interact with artificial intelligence and it’s likely that you’ve come into contact with it recently. Have you started a conversation with customer support on a website where your first point of contact was a chatbot? Sentiment analysis is what allows that bot to understand your responses and to point you in the right direction.

Have you tried translating something recently and wondered how the program is understanding your original? Well, if it works well, then that will be relying on Natural Language Processing (NLP) with sentiment analysis to help identify the contextual meaning and nuance of what you are trying to translate.

Sentiment analysis is all around us. So, if you’re new to the game and yet to start using it to your advantage, this article will help you to better understand its various applications and explain how you can start using sentiment analysis to gain invaluable business insights.

Not all sentiment analysis applies the same level of analysis to text, nor does it have to. There are times when simply understanding the polarity of a piece of text is sufficient, but most use cases (a topic that we go into in a later section) for sentiment analysis require natural language processing that is capable of identifying the emotional context of the text.

Sentiment analysis is very much a long-tail problem. That is to say that there are many different scenarios, subtleties, and nuances that can impact how a sentence is processed.

If you’re only concerned with the polarity of text, then your sentiment analysis will rely on a grading system to analyze your text. This might be sufficient and most appropriate for use cases where you are processing relatively simple sentences or multiple choice answers to surveys or feedback.

This graded sentiment analysis requires that you train your AI model to associate a certain score to different words and terms. From there, it can apply a plus or minus grading system to each piece of text to determine if it is positive, negative, or neutral. 

For example, you might train your AI model to associate the word fast as being positive and the phrase time-consuming as being negative. This means that the following sentence: 

The delivery time was really fast

Will be labelled as positive. Whereas this sentence: 

‍Filling in your return form was really time-consuming

Would be determined to be negative. These are both pretty simple examples, but even for polarity of text you might need a more powerful NLP once you start to deal with sentences that contain multiple sentiments. Say, for example: 

Filling in your return form was really time-consuming, but the refund was handled very quickly.

This sentence contains both positive and negative statements. If your AI model is insufficiently trained or your NLP is overly simplistic, then you run the risk that the analysis latches on to either the start or the end of the statement and only assigns it a single label.

Fortunately, with a platform like Lettria you can ensure that your NLP can identify multiple sentiments from within the same statement. But what if you want something a bit more complex than just polarity of text and you actually want to get into real emotion detection and analysis?

Emotion detection systems are a bit more complicated than graded sentiment analysis and require a more advanced NLP and a better trained AI model. 

Emotion detection requires more complex machine learning algorithms or detection systems that use lexicons with assigned emotional meanings or associations (in many ways, these detection systems are not that dissimilar to the approach used in graded sentiment analysis).

This obviously presents a number of monumental challenges and understanding and interpreting the emotional meaning behind a piece of text is not easy. Even humans make mistakes when it comes to analyzing the sentiment within text or speech, so training an AI model to do it accurately is not easy. 

{{ligne}}

Challenges to sentiment analysis

Language is an incredibly complicated thing. Just in writing this article I’ve managed to confuse myself on several occasions — and that’s when I’m faced with the relatively simple challenges of analyzing my own text. 

As we have already discussed, an NLPs AI model has to be fairly advanced in order to begin to identify the sentiment and emotional message expressed within a text. Some sentences are relatively straightforward, but the context and nuance of other phrases can be incredibly challenged to analyze. 

Take for example: 

The hotel has comfortable beds.

It would be relatively easy to label this sentence as positive. There’s a single adjective, it is clearly associated with a positive sentiment, and the sentence only conveys one message with no great complexity. 

But things can get complicated very quickly…

Context

The first challenge is to understand the context of the sentence. The meaning of the same set of words can vary greatly depending on the context in which they are said. It could be impacted by the previous sentence or the specifics of certain technical language.

Take for example, the following customer response:

Every single feature

If your support agent has just asked a customer what they like about your product then this is clearly a positive sentence, but if they just asked your customer what they didn’t like about your product then it would fall firmly into the negative category.

That’s why it’s important that your NLP is capable of not only analyzing the individual statements, sentences, and words, but also being able to understand their placement and usage from a contextual standpoint.

Figures of Speech

Figures of speech can also greatly change how sentences and words should be interpreted. The most obvious examples are with irony and sarcasm, where their presence can completely flip the meaning of a word or phrase. 

Let’s take another example from an exchange between a support agent and a customer. Say the support agent asks the customer if they are enjoying their experience with your product and they get the following response:

Well I’m here speaking to your customer support, so you can tell that I’m really loving it.

The contextual clues within the sentence and previous exchanges would allow an NLP capable of sentiment analysis to identify this statement as being sarcastic and so not label the sentence as positive, even though it would appear to be so when viewed in isolation.

Comparisons

Another area where sentiment analysis can ensure that natural language processing delivers the correct analysis is in situations where comparisons are being made. 

Comparisons can sometimes be simple and straightforward. Take the following examples: 

‍This product is better than any of the competition

This was the best experience that I’ve ever had 

Both statements are clearly positive and there’s no real requirement for any great contextual understanding. 

But things can get more complicated in a situation like this:

I suppose it is better than nothing

The statement would appear positive without any context, but it is likely to be a statement that you would want your NLP to classify as neutral, if not even negative. Situations like that are where your ability to train your AI model and customize it for your own personal requirements and preferences becomes really important. 

Lettria allows users to get their project up and running and customize their AI model 75% faster than the off-the-shelf NLPs. This means that not only are you able to start seeing results more quickly, but you’re also able to dedicate more time to really defining how you want your AI model to analyze the text and allow it to perform more accurate sentiment analysis by providing it with more examples and more clearly defining positive, negative, and neutral statements. 

That additional information can make all the difference when it comes to allowing your NLP to understand the contextual clues within the textual data that it is processing.

So, on that note, we’ve gone over the basics of sentiment analysis, but now let’s take a closer look at how Lettria approaches the problem.

{{ligne}}

‍How does Lettria handle sentiment analysis?

We’ve already hinted at the fact that not all NLPs are created equal, and Lettria has put itself into a unique category by providing users with a low-code or no-code platform that specializes in customizable textual data processing.

This platform-based approach allows users to benefit from Lettria’s 15 pre-trained and customizable multilingual models to employ an NLP to start with a stable foundation and focus on training their AI model to be highly specialized for their specific business, industry, and use case. 

Lettria offers all of the benefits of an off-the-shelf NLP (implementation and production time) with the power and customization of building one your own (but 4 times faster). Alright, that’s the sales pitch done, now let’s take a closer look at how Lettria actually handles sentiment analysis. 

Lettria’s API uses resources from psychology and the 8 primary emotions modelled in Putichik’s wheel of emotions (joy, sadness, fear, anger, attract, surprise, and anticipation). From there, we break the analysis into two steps. 

‍Step 1

Our algorithm analyzes the text to identify the adverbs and adjectives that are modifiers of meaning within a text. Once this is complete and a sentiment is detected within each statement, the algorithm then assigns a source and target to each sentence.

The emotional value of a statement is determined by using the following graded analysis.

Values are calculated by either using the sentiment objects or, if they are not available, by a prediction model at the sub sentence level. Values are normalized to stay within the -1 : 1 interval between the element, sub sentence and sentence level comparisons should (again) therefore be made with elements of the same depth.

With the sentiment of the statement being determined using the following graded analysis.

All sound a bit confusing? I know how you feel, but let’s use a real-world example to make things a bit clearer.

Take a simple sentence like ‘I like reading’ (at least, I hope you do if you’ve decided to make your way through this article).

The statement contains an overall positive sentiment, an emotion of joy as defined by the 8 primary emotions, and an emotional intensity of .46 (on a scale of -1 to 1). 

Now, say you’re really enjoying this article and decide to leave a comment saying ‘I really like reading’ then you would still return a positive sentence, but the addition of ‘really’ would increase the value of the emotion to .66. 

This first step essentially allows Lettria to carry out the graded sentiment analysis and polarity of text analysis that we discussed in the previous section. The second step is where we start to process the context and the real emotion expressed within the text. 

Step 2

To gain a more complete understanding of the emotions of a sentence, Lettria uses deep learning to identify the context of the sentiments within a text. This is done using the framework of 28 different emotional values. 

These emotional guidelines help the AI model to understand the context of the sentiments being expressed. When you combine steps 1 and 2, Lettria is not only able to determine the polarity of a statement, but also the emotional context and value within a sentence. 

So, when we then get a slightly more complicated example that looks something like this:

Your product is so unreliable, why does it always fail so easily.

The analysis would be able to determine that this statement is still negative, in spite of the fact that you could have trained your AI model to associate ‘easily’ with being a positive. 

In many ways, you can think of the distinctions between step 1 and 2 as being the differences between old Facebook and new Facebook (or, I guess we should now say Meta). At first, you could only interact with someone’s post by giving them a thumbs up. Which essentially meant that you could only react in a positive way (thumbs up) or neutral way (no reaction).

But, they eventually introduced the ability to use a wide range of different emojis that allowed you to express a variety of different emotions and reactions. This meant that the original poster had to think a bit more deeply when they wanted to interpret your reaction to their post (and account for the possibility that you might have been sarcastic or ironic). 

This increase in a user’s ability to react to posts is actually an example of a use case where sentiment analysis can be used to help better understand large data sets of social media interactions (more on that later), but it is a good way of thinking about when you might want a more advanced NLP to process your sentiment analysis and deliver more nuanced results.

The reality is, for all of the use cases and applications that we are about to touch on, you need an NLP that is capable of doing more than just graded sentiment analysis. 

The Lettria platform has been specifically developed to handle textual data processing and offers advanced sentiment analysis. Delivering a high level of accuracy and the ability to customize your AI model to suit all of your specific business and industry requirements, Lettria is able to address all of the use cases where sentiment analysis is applied.

So let’s take a closer look at when and where organizations are typically using natural language processing sentiment analysis and you can see if there are applications that could help you to improve your performance and your understanding of your customers and business.

{{ligne}}

Sentiment analysis use cases

Although the applications for natural language processing sentiment analysis are far-reaching and varied, there are a few use cases in which the analysis is commonly applied. Sentiment analysis can help organizations to gain invaluable business insights in virtually any situation in which they are faced with the task of handling large quantities of unstructured textual data, but here are a few examples of areas where your company or project could immediately benefit from sentiment analysis. 

Social Media Monitoring

If you aren’t listening to your customers wherever they speak about you then you are missing out on invaluable insights and information. That means that social media platforms are areas where your leads, customers, or former customers will be sharing their honest opinions about your product and services. 

You might have a social media team and some social media managers who track conversations and try to engage with users on various platforms, but it’s unlikely that they have the time, resources, and capabilities to not only track and interact with every conversation, but also accurately record the exchanges. 

That’s where natural language processing with sentiment analysis can ensure that you are extracting every bit of possible knowledge and information from social media.

Your knowledge and understanding of your leads and customers can be significantly increased by analyzing their social media posts and comments and, perhaps more importantly, real-time social media monitoring can help you to manage by exception and identify the comments and trends that require the most attention.

Research from McKinsey shows that customers spend 20 to 40 percent more with companies that respond on social media to customer service requests. Not only that, but companies that fail to respond to their customers on social media experience a 15 percent higher churn rate.

The even more concerning statistic for unresponsive companies? Customers expect to hear back from you quickly. 40 percent of consumers expect a response within the first hour and 79 expect a response within the first day. 

What does all of that mean? You risk losing business, and lots of it, if you’re not able to identify the social media posts and comments that require your attention and meaningful attention. 

Employing natural language processing sentiment analysis to analyze your social media in real-time will ensure that your social media and customer support teams will be able to identify the customers that require immediate human attention.

It increases efficiency, improves resource allocation and time management, and, most importantly again, improves customer experience and brand loyalty.

Read more about social media sentiment analysis here.

Customer Service

We’ve already touched on how sentiment analysis can improve your customer service on social media, but it can also improve your customer service performance through other channels. 

For starters, natural language processing sentiment analysis is a key element for high-performing chatbots. You may be employing an off-the-shelf chatbot that applies basic filters to your customer conversations, but you also have the ability to train an AI model that will be customized for your specific business needs and language.

What’s more, sentiment analysis can help you to filter incoming customer support tickets and ensure that they are labelled correctly, passed on to the appropriate team or department, and assigned the correct level of urgency.

Much like social media monitoring, this can greatly reduce the frustration that is often the result of slow response times when it comes to customer complaints. It is also another example of where sentiment analysis can help you to improve resource allocation and efficiency.

Learn more about how to improve customer service with sentiment analysis.

‍Brand Reputation

Our first couple of use case examples help companies to improve their short-term performance in a way that can have a lasting impact on long-term results, but our next few examples are much more focused on long-term thinking, planning, and performance.

Social media monitoring and customer service responses can play a key role in improving brand loyalty, but it also helps you to identify the areas of your brand that are performing the best and those that require attention.

Sentiment analysis allows you to train an AI model that will look out for thoughts and messages surrounding particular topics or areas. To monitor in real-time all of the conversations that relate to your brand and image. 

Brand loyalty is typically broken down into three key areas: 

• Perceived brand value - if your products or services offer competitive value within your industry
• Perceived brand quality - do your products or services match your competition when it comes to actual quality
• Perceived brand trust - do your customers feel comfortable with their decision to be loyal to your products or services

Understanding how your customers feel about each of these key areas can help you to reduce your churn rate. Research from Bain & Company has shown that increasing customer retention rates by as little as 5 percent can increase your profits by anywhere from 25 to 95 percent. 

We can all fall in love with the idea of a new customer, but making sure that you take care of your existing customers is just as important. Real-time monitoring through sentiment analysis will improve your understanding of your customers, help you to have more accurate net promoter scores, and ensure that your existing customers become loyal customers.

Market Research

There’s a good chance that you’ve already run campaigns that have included surveys and other initiatives to help you get feedback from leads and customers. That kind of market research can provide invaluable insights into your company, how you’re perceived, and your competition; however, if those interactions involve anything more complex than multiple choice answers then there’s a good chance that you’ve left some knowledge on the table. 

As with social media and customer support, written answers in surveys, product reviews, and other market research are incredibly time consuming to manually process and analyze. Natural language processing sentiment analysis solves this problem by allowing you to pay equal attention to every response and review and ensure that not a single detail is overlooked. It possible to analyse survey data with NLP emotion analysis.

Why put all of that time and effort into a campaign if you’re not even capable of really taking advantage of all of the results? Sentiment analysis allows you to maximize the impact of your market research and competitive analysis and focus resources on shaping the campaigns themselves and determining how you can use their results.

Voice of Customer and Net Promoter Scores

The brand reputation use case made mention of how sentiment analysis can help you to have a more accurate net promoter score, but it’s worth taking a closer look at how it can improve your understanding of your NPS and Voice of Customer (VoC).

Similar to the market research use case, if you’re interested in tracking your net promoter score than sentiment analysis may well be the only way that you can carry out large-scale analysis without overlooking some of your feedback and results. 

Open-ended questions have long been a nightmare for surveys and feedback, but sentiment analysis solves this problem by allowing you to process every bit of textual data that you receive.

Sentiment analysis will not only help you to get every bit of detail, knowledge, and information out of all of these responses, but real-time tracking will mean that you’ll no longer have to wait weeks, months, or potentially even years to be able to take that understanding and apply it to your business.

As you can see, sentiment analysis can provide meaningful results for companies and organizations in virtually any sector or industry. It can improve your understanding of your business and customers and increase efficiency and performance.

So, the question isn’t really whether or not natural language processing and sentiment analysis could be useful for you. It’s simply a question of how you can make sure that your NLP project is a success and produces the best possible results.

{{ligne}}

Start your own sentiment analysis project

If you’ve made it this far then it’s fair to say that there’s a strong possibility that you’re interested in exploring the benefits that Lettria’s sentiment analysis could bring to your project or organization. It might be because you’re frustrated with your existing NLP project or you’re only beginning to explore the world of natural language processing. 

Lettria’s platform-based approach means that, unlike most NLPs, both technical and non-technical profiles can play an active role in the project from the very beginning. This means that your work will not suffer from the silo effect that is the undoing of many NLP projects. 

All too often, NLP projects are thought of as being the exclusive domain for data scientists and developers. It is true that they may play a crucial role in getting the project up and running, but most of the time it is other teams and profiles that benefit from the results and insights that natural language processing produces.

Lettria solved that problem by making the platform accessible to non-technical profiles and focusing on developing an approach that provides all of the power of a customizable NLP with the added benefits of increased collaboration and knowledge sharing.

Not only can your data scientists and developers work on an NLP that offers incredibly accurate and efficient results, but business profiles can be directly involved in the project itself and contribute their own knowledge and expertise and gain a better understanding of the insights that your analysis is revealing.

This engagement and ability for technical and non-technical profiles to collaborate and easily contribute to the same project increases buy-in, reduces failure rates (85% of NLP projects fail before they begin producing any meaningful results), and improves results.

So, whether you’re reading this as a data scientist or developer with an advanced understanding of natural language processing, or you’re a sales person or customer support specialist with absolutely no background in NLP platforms, Lettria is the platform for you.

‍