The Bayes Classifier: Building a Tweet Sentiment Analysis Tool

Machine learning, with all its math and complexity, can be daunting. We’ll explore a relatively accessible technique: the Naive Bayes Classifier.

With things moving a bit more slowly through the holiday season, we’re going to re-run some of our most popular posts from 2015. Enjoy!
Machine learning can be a daunting subject. It involves involved subjects, a lot of mathematics, and sometimes emergent behavior beyond the understanding of the original implementers. This post will explore one of the easier, and more useful, machine learning techniques out there: Naive Bayes Classification.

It has been shown that we humans are quite bad at predicting outcomes, especially when there’s prior evidence. The decimals seem to scramble our brains and confuse us. Our human biases also seem to prevent us from making accurate predictions. Never mind the fact that going through documents or text and classifying it can be tedious and time-consuming. Machines, on the other hand, are not biased, do not get confused by decimals, and can do calculations much quicker than we do.

Bayes Classifier: The mathematics

A naive Bayes classifier applies Bayes’ Theorem in an attempt to suggest possible classes for any given text. To do this, it needs a number of previously classified documents of the same type. The theorem is as follows:
Bayes Classifier: machine learning

Bayes Classifier example: tweet sentiment analysis

As an example, let us try and find the probability that a tweet (the document) can be classified as positive (the class). At first glance the theorem can be confusing, so let’s simplify it a bit by breaking down the various components:

P(A|B)
This can be read as the probability of A, the class, given B, the tweet. This is the end result we’re looking for.
P(B|A)
This can be read as the probability of B, the tweet, given A, the class. This is determined by previously gathered information.
P(A)
This is the probability of A – the class. It’s independent of all other probabilities.
P(B)
This is the probability of B – the tweet. It’s independent of all other probabilities.

Bayes Classifier: bayes theorem
Since the probability of the tweet, P(tweet), is constant, it can be disregarded in our calculations. We’re only interested in the probability of the tweet given the class, P(tweet|positive), and the probability of the class, P(positive):
P(positive|tweet) = P(tweet|positive) * P(positive)

P(positive)

For the sake of this example, let’s say there are three possible classes: positive, negative and neutral. That gives any tweet a one in three (or 33%) chance of falling into any of those classes. That gives us P(positive) = 0.33333.

P(tweet|positive)

To calculate P(tweet|positive), we need a training set of tweets that were already classified into three categories. This gives us a basis from which to compute the probability that a tweet will fall into a specific class. Since the chances are relatively low that we’ll find a specific tweet in the training set, we’ll tokenize the tweet and calculate the probability for each word in the training set. This gives us the following formula:
P(tweet|positive) = P(T1|positive) * P(T2|positive) * .. * P(Tn|positive)
Where T1 to Tn is all the words in the tweet.

P(T1|positive)

To determine the probability of a specific word falling into the category we’re testing, we’ll need the following from the training set:

  • The number of times T1 occurs in tweets that were marked as positive in the training set.
  • The total number of words of tweets that were marked as positive in the training set.

There are various ways in which you can get these numbers, so we won’t go into specifics here. As an example, let’s look at the word “food”, with the following numbers:

  • Number of times food occurs in positive tweets: 455
  • Number of words in positive tweets: 1211

So to calculate the relative probability of food occurring in the positive category, we divide 455 by 1211, giving us 0.376. Since food can have positive, negative and neutral interpretations, it’s not surprising that its relative probability is 37%. This process now needs to be repeated for each word in the tweet.

Since we now have the ability to calculate the probabilities that each word in the tweet can be classified as positive, let’s calculate the probability that the whole tweet can be classified as positive – P(positive|tweet) = P(tweet|positive) * P(positive). For this example, let’s say the tweet was “I love good food”, and the probabilities we calculated were 25%, 62.5%, , 4% and 42.5% respectively.
P(positive|tweet) = P(tweet|positive) * P(positive)
= P(T1|positive) * .. * P(Tn|positive) * P(positive)
= 0.25 * 0.625 * 0.74 * 0.425 * 0.33
= 0.016216406

This same procedure can now be used to calculate the relative probability for each of the classes. From the training set, we calculate P(negative|tweet) as 0.000003125 and P(neutral|tweet) as 0.0082809375. Once we have the probability for each class, we can compare the classes, and use the highest ranked class as the class for the document. Intuitively, it makes sense to classify I love good food as positive, but now we have a mathematical proof, based on gathered data, that it can be classified as positive.

Bayes Classifier: some considerations

When you read up on the Bayes classifier, you’ll see that it’s often called the Naive Bayes classifier. It’s called naive because the classifier assumes that the document and their words are independent of each other. This assumption greatly simplifies and at the same time speeds up the needed calculations, but reduces the classifier’s accuracy. Despite this reduced accuracy, the classifier is still surprisingly accurate, and fast to boot.

There are some features of the theorem or the data set that can severely skew the calculated probabilities. On the one hand, the repeated use of decimals can result in very low numbers, sometimes interpreted as zero, on computers. This is known as underflow. On the other hand, if we try to calculate the probability for a word that doesn’t exist in the training set, it will come out as zero. Since the final probability is the product of the probabilities of all the words, this will result in a final probability of zero as well, regardless of how high, or low, the other probabilities are. To prevent this from happening, we apply a technique called smoothing. Using these techniques will greatly increase the accuracy of your classifier.

Bayes Classifier: implementation

It’s relatively easy to find an implementation of the Bayes classifier in your language of choice. A couple of examples are the classifier gem for Ruby, and the NLP package for PHP. The code below shows the classification of the tweet we’ve just discussed using a previously defined training set and the classifier gem:

require 'classifier'
# Set up the classifier
classifier = Classifier::Bayes.new('Positive', 'Neutral', 'Negative')
# Train the classifier
CSV.foreach('training_set.csv') do |row| # In the format category,tweet
  classifier.train(row[0], row[1])
end
# Use the classifier
b.classify 'I love good food' # Returns "Positive"

You may also find this dataset useful for experimenting on your own.

Summary

Despite all the complicated mathematics, implementing a Bayes classifier is all about counting the number of words, documents and categories. Once you have these, you can combine them to calculate the probability for each of the possible classes. The document is then classified according to the highest calculated probability. Although there are some factors to take into consideration when using the Bayes filter, in general, it should prove to be a profitable and easy first step into Machine Learning.

If you’d like to understand more Machine Learning, visit Cloud Academy’s Machine Learning training library.

Avatar

Written by

Jurgens du Toit

A Full Stack Developer with a great interest in DevOps and Data Analytics. I can code anything from an API to a single page app, as well as deploy and manage a system and it's infrastructure using tools such as Ansible and Chef. Further to that I like working with data. Analyzing, wrangling and interpreting data until it becomes information is a process I find very interesting.


Related Posts

Alisha Reyes
Alisha Reyes
— July 2, 2020

New Content: AWS, Azure, Typescript, Java, Docker, 13 New Labs, and Much More

This month, our Content Team released a whopping 13 new labs in real cloud environments! If you haven't tried out our labs, you might not understand why we think that number is so impressive. Our labs are not “simulated” experiences — they are real cloud environments using accounts on A...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Alisha Reyes
Alisha Reyes
— June 11, 2020

New Content: AZ-500 and AZ-400 Updates, 3 Google Professional Exam Preps, Practical ML Learning Path, C# Programming, and More

This month, our Content Team released tons of new content and labs in real cloud environments. Not only that, but we introduced our very first highly interactive "Office Hours" webinar. This webinar, Acing the AWS Solutions Architect Associate Certification, started with a quick overvie...

Read more
  • AWS
  • Azure
  • DevOps
  • Google Cloud Platform
  • Machine Learning
  • programming
Vijayakumar Athithan
Vijayakumar Athithan
— May 8, 2020

OWASP Top 10 Vulnerabilities

Over the last few years, more than 10,000 Open Web Application Security Project (OWASP) vulnerabilities have been reported into the Common Vulnerabilities and Exposures (CVE®) database each year. This is a list of common identifiers for publicly known cybersecurity vulnerabilities. Curr...

Read more
  • Machine Learning
  • OWASP
  • OWASP Top 10
  • Security
Alice Jones
Alice Jones
— February 28, 2020

AI and Machine Learning: How They Are Changing the Content Industry

Machine learning falls under an array of artificial intelligence (AI) technologies that learn how to do certain tasks with the intention of automating them. These systems use historical data to predict future patterns and execute their tasks according to accurate data gathered. The more...

Read more
  • AI
  • Artificial Intelligence
  • content marketing
  • Machine Learning
  • ML
Avatar
Andrew Larkin
— February 13, 2020

Cloud Academy Content Roadmap Updates

Welcome to our Q1 2020 roadmap. This is the content we plan to build over the next three months, between February 1 - and April 30, 2020. Let's look at some of our roadmap highlights. Atlassian Bamboo for CI/CD We had a lot of requests for practical guides on how to apply DevOps tool...

Read more
  • Artificial Intelligence
  • AWS
  • Azure
  • Docker
  • Google Cloud Platform
  • Kubernetes
  • Machine Learning
Avatar
Stuart Scott
— July 2, 2019

AWS Machine Learning Services

The speed at which machine learning (ML) is evolving within the cloud industry is exponentially growing, and public cloud providers such as AWS are releasing more and more services and feature updates to run in parallel with the trend and demand of this technology within organizations t...

Read more
  • Amazon Machine Learning
  • AWS
  • AWS re:Invent
  • Machine Learning
Francesco Mosconi
Francesco Mosconi
— May 27, 2019

What is Deep Learning and Does Your Enterprise Need It?

What is Deep Learning? The most frequent question asked by my students is: Do I need to learn deep learning? Beyond the buzzwords bounced back and forth in blog posts and news articles, deep learning is probably the most revolutionary technology of the last century. Discovered in the ...

Read more
  • Machine Learning
Luca Casartelli
Luca Casartelli
— April 19, 2019

4 Key Takeaways from Google Cloud Next ’19

Google Cloud Next ’19 was the flagship Google Cloud Platform developers conference, held in San Francisco’s Moscone Center. I was lucky enough to attend it with Cloud Academy, and got the chance to check out tons of breakout sessions and get great insight firsthand.  Next ’19 was my...

Read more
  • Google Cloud Platform
  • Kubernetes
  • Machine Learning
Avatar
David Santucci
— February 14, 2019

How to Build an Intelligent Chatbot with Python and Dialogflow

Chatbots are a powerful example of artificial intelligence (AI) in use today. Just think about Google Assistant and how intelligent the platform became thanks to machine learning. But, what is a chatbot? How do you create a custom bot for your website? Which technologies can you use to ...

Read more
  • Machine Learning
  • Python
Avatar
Dwayne Monroe
— January 8, 2019

What is Azure Machine Learning

The meal was fantastic, the service was friendly and professional, the setting was cozy, and the company was engaging. As the evening ended, however, there was a slight hiccup as my credit card was declined. There was more than enough money in my account to cover the cost of the (very d...

Read more
  • Machine Learning
Albert Qian
Albert Qian
— September 25, 2018

Microsoft Ignites Cloud Industry With Nadella Keynote

On Monday, Microsoft kicked off its Ignite conference, an annual gathering of developers and IT professionals. Over the next week, attendees will learn about upcoming Microsoft innovations in IoT, artificial intelligence, machine learning, and cloud (all while getting some good networki...

Read more
  • Events
  • IoT
  • Machine Learning
  • Security
Avatar
Guy Hummel and Jeremy Cook
— August 23, 2018

What are the Benefits of Machine Learning in the Cloud?

A Comparison of Machine Learning Services on AWS, Azure, and Google Cloud Artificial intelligence and machine learning are steadily making their way into enterprise applications in areas such as customer support, fraud detection, and business intelligence. There is every reason to beli...

Read more
  • AWS
  • Azure
  • Google Cloud Platform
  • Machine Learning