保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Probability

Text Classification Using Naive Bayes: Theory & A Working Example

In this article, I explain how the Naive Bayes works and I implement a multi-class text classification problem step-by-step in Python. Serafeim Loukas, PhD October 12, 202010 min read

Getting Started

Figure created by the author. Figure created by the author. Learn this step by step with the interactive Computer Science and Machine Learning roadmaps.

Table of contents

  1. Introduction
  2. The Naive Bayes algorithm
  3. Dealing with text data
  4. Working Example in Python (step-by-step guide)
  5. Bonus: Having fun with the model
  6. Conclusions

1. Introduction

Naive Bayes classifiers are a collection of classification algorithms based on Bayes' Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being classified is independent of each other. Naive Bayes classifiers have been heavily used for text classification and text analysis machine learning problems. Text Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols (i.e. strings) cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length. In this article I explain a) how Naive Bayes works**,** b) how we can use text data and fit them into a model after transforming them into a more appropriate form. Finally, I implement a multi-class text classification problem step-by-step in Python. Let's get started !!!

2. The Naive Bayes algorithm

Naive Bayes classifiers are a collection of classification algorithms based on Bayes' Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being classified is independent of each other. The dataset is divided into two parts, namely, feature matrix and the response/target vector.

2.1. The main two assumptions of Naive Bayes

Naive Bayes assumes that each feature/variable of the same class makes an:

contribution to the outcome. Side Note: The assumptions made by Naive Bayes are not generally correct in real-world situations. In-fact, the independence assumption is often not met and this is why it is called "Naive " i.e. because it assumes something that might not be true.

2.2. The Bayes' Theorem

Bayes' Theorem finds the probability of an event occurring given the probability of another event that has already occurred. Bayes' theorem is stated mathematically as follows: Figure created by the author. Figure created by the author. where:

Summary

Figure created by the author. Figure created by the author.

2.3. The Naive Bayes Model

Given a data matrix X and a target vector y, we state our problem as: Figure created by the author. Figure created by the author. where, y is class variable and X is a dependent feature vector with dimension d i.e. X = (x1,x2,x2, xd), where d is the number of variables/features of the sample.

Now the "naïve" conditional independence assumptions come into play: assume that all features in X are mutually independent, conditional on the category y : Figure created by the author. Figure created by the author. Finally, to find the probability of a given sample for all possible values of the class variable y , we just need to find the output with maximum probability: Figure created by the author. Figure created by the author.

3. Dealing with text data

One question that arises at this point is the following:

How are we going to use the raw text data to train the model ? The raw data is a collection of strings ! Text Analysis is a major application field for machine learning algorithms. However the raw data, a sequence of symbols (i.e. strings) cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length. In order to address this, scikit-learn provides utilities for the most common ways to extract numerical features from text content, namely:

In this scheme, features and samples are defined as follows:

"Counting" Example (to really understand this before we move on):

text

from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]

text

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)

text

print(vectorizer.get_feature_names())
['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']

text

print(X.toarray())
[[0 1 1 1 0 0 1 0 1]
 [0 2 0 1 0 1 1 0 1]
 [1 0 0 1 1 0 1 1 1]
 [0 1 1 1 0 0 1 0 1]]

In the above toy example, we have a collection of strings stored into the variable corpus. Using the text transformer , we can see that we have a specific number of unique strings (vocabulary) in our data. This can be seen by printing the vectorizer.get_feature_names() variable**.** We observe that we have 9 unique words. Next, we printed the transformed data (X) and we observe the following :

E.g. The first transformed row is [0 1 1 1 0 0 1 0 1] and **** th** e unique vocabularyi s ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this'],** thus this means that the words "document", "first", "is", "the" and "this" appeared 1 time each in the initial text string (i.e. 'This is the first document.'). Side note: This is the counting approach. The term-frequency transform is nothing more than a transformation of a count matrix into a normalized term-frequency matrix. Hope everything is clear now. If not, read this paragraph as many times as it is needed in order to really grasp the idea and understand this transformation. It is a fundamental step.

4. Working example in Python

Now that you understood how the Naive Bayes and the Text Transformation work, it's time to start coding !

Problem Statement

As a working example, we will use some text data and we will build a Naive Bayes model to predict the categories of the texts. This is a multi-class (20 classes) text classification problem. Let's start (I will walk you through). First, we will load all the necessary libraries : text

import numpy as np, pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline
from sklearn.metrics import confusion_matrix, accuracy_score

text

sns.set() # use seaborn plotting style

Next, let's load the data (training and test sets): text

# Load the dataset
data = fetch_20newsgroups()

text

# Get the text categories
text_categories = data.target_names

text

# define the training set
train_data = fetch_20newsgroups(subset="train", categories=text_categories)

text

# define the test set
test_data = fetch_20newsgroups(subset="test", categories=text_categories)

Let's find out how many classes and samples we have: text

print("We have {} unique classes".format(len(text_categories)))
print("We have {} training samples".format(len(train_data.data)))
print("We have {} test samples".format(len(test_data.data)))

The above prints: text

We have 20 unique classes
We have 11314 training samples
We have 7532 test samples

So, this is a 20-class text classification problem with n_train = 11314 training samples (text sentences) and n_test = 7532 test samples (text sentences). Let's visualize the 5th training sample: text

# let's have a look as some training data
print(test_data.data[5])

As mentioned previously, our data are texts (more specifically, emails) so you should see something like the following printed out: Figure created by the author. Figure created by the author. The next step consists of building the Naive Bayes classifier and finally training the model. In our example, we will convert the collection of text documents (train and test sets) into a matrix of token counts (I explained how this works in Section 3).

To implement that text transformation we will use the make_pipeline function. This will internally transform the text data and then the model will be fitted using the transformed data. text

# Build the model
model = make_pipeline(TfidfVectorizer(), MultinomialNB())

text

# Train the model using the training data
model.fit(train_data.data, train_data.target)

text

# Predict the categories of the test data
predicted_categories = model.predict(test_data.data)

The last line of code predicts the labels of the test set. Let's see the predicted categories names: text

print(np.array(test_data.target_names)[predicted_categories])
array(['rec.autos', 'sci.crypt', 'alt.atheism', ..., 'rec.sport.baseball', 'comp.sys.ibm.pc.hardware', 'soc.religion.christian'], dtype='<U24')

Finally, let's build the multi-class confusion matrix to see if the model is good or if the model predicts correctly only specific text categories. text

# plot the confusion matrix
mat = confusion_matrix(test_data.target, predicted_categories)
sns.heatmap(mat.T, square = True, annot=True, fmt = "d", xticklabels=train_data.target_names,yticklabels=train_data.target_names)
plt.xlabel("true labels")
plt.ylabel("predicted label")
plt.show()

text

print("The accuracy is {}".format(accuracy_score(test_data.target, predicted_categories)))

text

The accuracy is 0.7738980350504514

Figure created by the author. Figure created by the author.

Note: If you're passionate about diving deeper into machine learning with Python and sklearn, I highly recommend checking out this book; it's a game-changer in breaking down complex topics into digestible insights.

Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques...

5. Bonus: Having fun with the model

Let's have some fun using the trained model. Let's classify whatever sentence we like 😄 . text

# custom function to have fun
def my_predictions(my_sentence, model):
    all_categories_names = np.array(data.target_names)
    prediction = model.predict([my_sentence])
    return all_categories_names[prediction]

text


my_sentence = "jesus"
print(my_predictions(my_sentence, model))
['soc.religion.christian']

text

my_sentence = "Are you an atheist?"
print(my_predictions(my_sentence, model))
['alt.atheism']

We inserted the string "jesus" to the model and it predicted the class "['soc.religion.christian']". Change the "my_sentence" into other stings and play with the model 😃 .

6. Conclusions

We saw that Naive Bayes is a very powerful algorithm for multi-class text classification problems. Side Note : If you want to know more about the confusion matrix (and the ROC curve) read this:

ROC Curve Explained using a COVID-19 hypothetical example: Binary & Multi-Class Classification...

Interpreting the confusion matrix

From the above confusion matrix , we can verify that the model is really good.

Interesting thing that these 2 categories are really similar and actually one could characterize these as 2 subgroups of a larger group e.g. "religion" in general. Finally, the accuracy on the test set is 0.7739 which is quite good for a 20-class text classification problem 🚀 . That's all folks! Hope you liked this article.

If you liked and found this article useful, follow 👣 me to be able to see all my new posts. Questions? Post them as a comment and I will reply as soon as possible.

My Profile (have a look to find out about my collection of articles):

Serafeim Loukas - Towards Data Science

Get in touch with me

You may also like:

Support Vector Machines (SVM) clearly explained: A python tutorial for classification problems... K-Means Clustering: How It Works & Finding The Optimum Number Of Clusters In The Data LSTM Time-Series Forecasting: Predicting Stock Prices Using An LSTM Model Time-Series Forecasting: Predicting Stock Prices Using An ARIMA Model The Best FREE Data Science Resources: FREE Books & Online Courses ROC Curve Explained using a COVID-19 hypothetical example: Binary & Multi-Class Classification... Support Vector Machines (SVM) clearly explained: A python tutorial for classification problems... PCA clearly explained - How, when, why to use it and feature importance: A guide in Python Everything you need to know about Min-Max normalization in Python How Scikit-Learn's StandardScaler works

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Serafeim Loukas, PhD

See all from Serafeim Loukas, PhD ClassificationGetting StartedMachine LearningNaive BayesProbability Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

How do Machine Learning and Signal Processing Blend?

Machine Learning An Overview Of IBM's "Advanced Machine Learning and Signal Processing" on Coursera Keenan Kon August 2, 20209 min read

The Machine Learning “Advent Calendar” Day 9: Local Outlier Factor in Excel

Artificial Intelligence LOF explained step by step, with tiny datasets Angela and Kezhan Shi December 9, 20257 min read

Downloading Datasets from Kaggle for Your ML Project

Machine Learning How to leverage the Kaggle Python API to download any dataset from their website Heiko Hotz January 6, 20223 min read

Step by Step Identity Resolution With Python And Zingg

Machine Learning An open source, machine learning based approach to resolve identities at scale Sonal Goyal November 22, 202211 min read

Under the hood - how do neural networks really work?

Artificial Intelligence Understanding how the math behind neural networks using the MNIST dataset - my key takeaways from fast.ai Daniel Ching November 28, 20213 min read

Machine Learning Model as a Serverless Endpoint using Google Cloud Functions

Artificial Intelligence So you have built a model and want to productionize it as a serverless solution on google cloud platform (GCP). Let me show you how to do... Saed Hussain January 17, 20216 min read

The Hardest Part: Defining A Target For Classification

Machine Learning It isn't labeled 'Target_Variable' in your Production Database! Chris Bruehl November 14, 20235 min read

Python Package for Bayesian Testing

Probability Quick evaluation of A/B (or A/B/C/...) tests using Bayesian approach Matúš Baniar January 3, 20223 min read

Fighting Back Against Attacks in Federated Learning

Machine Learning Lessons from a multi-node simulator Salman Toor September 10, 20257 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close