保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Machine Learning

Practical Introduction to Transformer Models: BERT

Hands-on tutorial on how to build your first sentiment analysis model using BERT Shashank Kapadia July 17, 20236 min read A coffee cup with white liquid and a coffee cup with dark liquid on contrasting backgrounds Photo by Alex Padurariu on Unsplash Preface: This article presents a summary of information about the given topic. It should not be considered original research. The information and code included in this article may have been influenced by things I have read or seen in the past from various online articles, research papers, books, and open-source code.

Table of contents

In NLP, the transformer model architecture has been a revolutionary model architecture that greatly enhanced the ability to understand and generate textual information. In this tutorial, we are going to dig deep into BERT, a well-known transformer-based model, and provide an hands-on example to fine-tune the base BERT model for sentiment analysis.

Introduction to BERT

BERT, introduced by researchers at Google in 2018, is a powerful language model that uses transformer architecture. Pushing the boundaries of earlier model architecture, such as LSTM and GRU, that were either unidirectional or sequentially bi-directional, BERT considers context from both past and future simultaneously. This is due to the innovative “attention mechanism,” which allows the model to weigh the importance of words in a sentence when generating representations. Learn this step by step with the interactive Machine Learning roadmap. The BERT model is pre-trained on the following two NLP tasks:

and is generally used as the base model for various downstream NLP tasks, such as sentiment analysis, which we will cover in this tutorial.

Pre-training and fine-tuning

The power of BERT comes from its two-step process:

Hands on: Using BERT for sentiment analysis

The complete code is available as a Jupyter Notebook on GitHub In this hands-on exercise, we will train the sentiment analysis model on the IMDB movie reviews dataset [4] (license: Apache 2.0), which comes labeled whether a review is positive or negative. We will also load the model using the Hugging Face’s transformers library. Let’s load all the libraries: python

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, roc_curve, auc
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer

# Variables to set the number of epochs and samples
num_epochs = 10
num_samples = 100  # set this to -1 to use all data

First, we need to load the dataset and the model tokenizer. python

# Step 1: Load dataset and model tokenizer
dataset = load_dataset('imdb')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

Next, we’ll create a plot to see the distribution of the positive and negative classes. python

# Data Exploration
train_df = pd.DataFrame(dataset["train"])
sns.countplot(x='label', data=train_df)
plt.title('Class distribution')
plt.show()

Fig 1. Class distribution of the training dataset Next, we preprocess our dataset by tokenizing the texts. We use BERT’s tokenizer, which will convert the text into tokens that correspond to BERT’s vocabulary. python

# Step 2: Preprocess the dataset
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)

After that, we prepare our training and evaluation datasets. Remember, if you want to use all the data, you can set the num_samples variable to -1. python

if num_samples == -1:
    small_train_dataset = tokenized_datasets["train"].shuffle(seed=42)
    small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42)
else:
    small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(num_samples)) 
    small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(num_samples))

Then, we load the pre-trained BERT model. We’ll use the AutoModelForSequenceClassification class, a BERT model designed for classification tasks.

For this tutorial, we use the ‘bert-base-uncased’ version of BERT, which is trained on lower-case English text, is used for this tutorial. python

# Step 3: Load pre-trained model
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

Now, we’re ready to define our training arguments and create a Trainerinstance to train our model. python

# Step 4: Define training arguments
training_args = TrainingArguments("test_trainer", evaluation_strategy="epoch", no_cuda=True, num_train_epochs=num_epochs)

# Step 5: Create Trainer instance and train
trainer = Trainer(
    model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)

trainer.train()

Interpreting results

Having trained our model, let’s evaluate it. We’ll calculate the confusion matrix and the ROC curve to understand how well our model performs. python

# Step 6: Evaluation
predictions = trainer.predict(small_eval_dataset)

# Confusion matrix
cm = confusion_matrix(small_eval_dataset['label'], predictions.predictions.argmax(-1))
sns.heatmap(cm, annot=True, fmt='d')
plt.title('Confusion Matrix')
plt.show()

# ROC Curve
fpr, tpr, _ = roc_curve(small_eval_dataset['label'], predictions.predictions[:, 1])
roc_auc = auc(fpr, tpr)

plt.figure(figsize=(1.618 * 5, 5))
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.show()

Fig 2. Confusion Matrix Fig 3. ROC curve The confusion matrix gives a detailed breakdown of how our predictions measure up to the actual labels, while the ROC curve shows us the trade-off between the true positive rate (sensitivity) and the false positive rate (1 — specificity) at various threshold settings. Finally, to see our model in action, let’s use it to infer the sentiment of a sample text. python

# Step 7: Inference on a new sample
sample_text = "This is a fantastic movie. I really enjoyed it."
sample_inputs = tokenizer(sample_text, padding="max_length", truncation=True, max_length=512, return_tensors="pt")

# Move inputs to device (if GPU available)
sample_inputs.to(training_args.device)

# Make prediction
predictions = model(**sample_inputs)
predicted_class = predictions.logits.argmax(-1).item()

if predicted_class == 1:
    print("Positive sentiment")
else:
    print("Negative sentiment")

Closing thoughts

By walking through an example of sentiment analysis on IMDb movie reviews, I hope you’ve gained a clear understanding of how to apply BERT to real-world NLP problems. The Python code I’ve included here can be adjusted and extended to tackle different tasks and datasets, paving the way for even more sophisticated and accurate language models.

References

[1] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 [2] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998–6008). [3] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., … & Rush, A. M. (2019). Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. [4] Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., Davison, J., Šaško, M., Chhablani, G., Malik, B., Brandeis, S., Le Scao, T., Sanh, V., Xu, C., Patry, N., McMillan-Major, A., Schmid, P., Gugger, S., Delangue, C., Matussière, T., Debut, L., Bekman, S., Cistac, P., Goehringer, T., Mustar, V., Lagunas, F., Rush, A., & Wolf, T. (2021). Datasets: A Community Library for Natural Language Processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 175–184). Online and Punta Cana, Dominican Republic: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2021.emnlp-demo.21 Thanks for reading. If you have any feedback, please feel to reach out by messaging me on LinkedIn or shooting me an email (smhkapadia[at]gmail.com).

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

SK Written By

Shashank Kapadia

See all from Shashank Kapadia BertHugging FaceNLPTransformer ModelTransformers Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Machine Learning and Analysis of Site Position Data

Machine Learning What are Google's machine learning algorithms, and how do they help to get higher positions in SERP Diana Ford August 17, 20216 min read

Hands-On Global Optimization Methods, with Python

Machine Learning Four methods to find the maximum (or minimum) of your black box objective function Piero Paialunga September 4, 202413 min read

Analysis of Toronto Neighbourhoods using Machine Learning

Machine Learning A New Immigrant's Guide to settling in the City of Toronto Jessica Ayodele November 27, 20205 min read

Dimensionality Reduction Made Simple: PCA Theory and Scikit-Learn Implementation

Machine Learning Tame the Curse of Dimensionality! Learn Dimensionality Reduction (PCA) and implement it with Python and Scikit-Learn. Riccardo Andreoni February 7, 202411 min read

Monitoring of an Underwater Drone

Machine Learning Monitoring streaming data with Kafka and whylogs The importance of data in making informed decisions... Felipe Adachi May 4, 202115 min read

The Word2vec Classifier

Machine Learning How word embeddings are trained Jon Gimpel July 26, 202215 min read

What is the Best Facial Recognition Software to Use in 2022?

Artificial Intelligence I checked dozens of free and paid facial recognition services. Here's what I learned Serhii Pospielov March 11, 202111 min read

kegra: Deep Learning on Knowledge Graphs with Keras

Artificial Intelligence Hi there. I mentioned in past articles that I am working intensely on cognitive computing for enterprise datasets. This is that. Daniel Shapiro, PhD December 5, 20177 min read

Word Embedding in NLP: One-Hot Encoding and Skip-Gram Neural Network

NLP I'm a poet-turned-programmer who has just begun learning about the wonderful world of natural language processing. In this post, I'll be... Jianna Park July 24, 202010 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close