保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search

Sentiment Analysis and Structural Breaks in Time-Series Text Data

Arabica now offers a structural break and sentiment analysis module to enrich time-series text mining Petr Koráb March 6, 20237 min read Photo by Adam Śmigielski on Unsplash Photo by Adam Śmigielski on Unsplash

Introduction

Text data contains lots of qualitative information, which can be quantified with various methods, including sentiment analysis techniques. These models are used to identify, extract and quantify emotions from text data and have wide use in business and academic research. Since the text is often recorded on a time-series basis, text datasets might display structural breaks as the quantitative information change due to many possible factors. As a business analyst, measuring the changes in customer perceptions of a particular brand might be one of the key tasks. In the research role, one can be interested in the shifts in Vladimir Putin's public statements over time. Arabica is a python library specifically designed to deal with similar questions. It contains these methods for exploratory analysis of time-series text datasets:

This article will introduce you to Coffee-break, the sentiment and structural breaks analysis module. Read the documentation and these tutorials for the first two methods: arabica_freq, cappuccino. EDIT Jul 2023 : Arabica has been updated. Check the documentation for the full list of parameters.

2. Coffee_break: algorithm and structure

The _coffee break module has a simple backend architecture. Here is schematically how it works: Figure 1. Coffe_break architecture. Source: draw.io Figure 1. Coffe_break architecture. Source: draw.io Raw text is cleaned with cleantext providing punctuation and numbers cleaning. Stop words (the most common words in a language with no significant meaning) are not removed in the pre-processing step because they don't negatively affect sentiment analysis. However, with skip parameter, we can remove a list of additional stop words or unwanted strings (words or word sequences) that will not impact sentiment analysis. Sentiment analysis implements VADER (Valence Aware Dictionary and Sentiment Reasoner ) , __ a universal pre-trained sentiment classifier [1]. It was trained on social media data from Twitter, but it also works very well on other types of datasets. My previous article offers a more detailed introduction to the model and coding in Python. Coffee_break uses VADER's compound indicator for sentiment evaluation. The aggregate sentiment is calculated as: where t is the aggregation period. The aggregate indicator ranges [-1: 1] having a positive sentiment closer to 1 and a negative approaching -1. The aggregated sentiment creates a time series displaying some degree of variability over time. Structural breaks in the time series are identified with the Fisher-Jenks algorithm, or Jenks Optimisation Method originally proposed by George F. Jenks [2]. It is a clustering-based method designed to find the best arrangement of values into different classes (clusters). The jenks_breaks function implemented with jenkspy library returns a list of values that correspond to the limits of the classes. These structural breaks are in the plot marked as vertical lines and visually indicate the breakpoints in the time series of text data. The implemented libraries are Matplotlib (visualization), vaderSentiment (sentiment analysis), and jenkspy (structural breaks). Pandas and Numpy make the processing operations.

3. Use case: Twitter sentiment analysis

Let's illustrate the coding on Pfizer Vaccine Tweets dataset collected using Twitter API. The data contains 11 000 tweets about Pfizer & BioNTech vaccine posted between 2006 and 2021. The dataset is released under the CC0: Public Domain license according to Twitter developer policy. The data contains a lot of punctuation and numbers and needs cleaning before any further steps: Figure 2. Pfizer Vaccine Tweets dataset Figure 2. Pfizer Vaccine Tweets dataset The _coffee break method's parameters are: python

def coffee_break(text: str,                 # Text column
                 time: str,                 # Time column
                 date_format: str,          # Date format: 'eur' - European, 'us' - American
                 time_freq: int ='',        # Aggregation period: 'Y'/'M'
                 preprocess: bool = False,  # Clean data from numbers and punctuation
                 skip: [] ,                 # Remove additional stop words
                 n_breaks: int =''          # Number of breaks: min. 2
)

3. 1. Sentiment analysis over time

Our data has a 15-year time span covering the Covid-19 crisis. Changes in the public mood about vaccination, fake news about vaccines, and many other factors are expected to lead to significant variations in sentiment over time.

Coding

First, import _coffee break : python

from arabica import coffee_break

Arabica reads dates in US-style (MM/DD/YYYY) and European-style (DD/MM/YYYY) date and datetime formats. The data is pretty raw and covers 15 years. Displaying sentiment by month is, therefore, not very helpful. Let's clean the data and aggregate sentiment by year with this code: python

coffee_break(text = data['text'],
             time = data['date'],
             date_format = 'eur',  # Read dates in European format
             time_freq = 'Y',      # Yearly aggregation
             preprocess = True,    # Clean data - punctuation + numbers
             skip = None ,         # No other stop words removed
             n_breaks = None)      # No structural break analysis

Results

Arabica returns a picture that can be manually saved in PNG or JPEG. Figure 3. Sentiment analysis - yearly Figure 3. Sentiment analysis - yearly At the same time, Arabica returns a dataframe with the corresponding data. The table can be saved simply by assigning the function to an object : python

# generate a dataframe
df = coffee_break(text = data['text'],
                  time = data['date'],
                  date_format = 'eur',
                  time_freq = 'Y',
                  preprocess = True,
                  skip = None ,
                  n_breaks = None)

# save is as a csv
df.to_csv('sentiment_data.csv')

Results interpretation: we can see that sentiment significantly dropped after Pfizer vaccines started to be used to tackle Covid in 2021 (Figure 2). The reason is likely the global pandemic the world faced and the generally negative mood in these years.

3.2. Structural break analysis

Next, let's formalize the structural breaks in sentiment statistically. _Coffe break enables the identification of min. 2 breakpoints. The following code returns a figure with 3 breakpoints marked by vertical lines and the table with the corresponding time series: python

coffee_break(text = data['text'],
             time = data['date'],
             date_format = 'eur', # Read dates in European format
             time_freq = 'Y',     # Yearly aggregation
             preprocess = True,   # Clean data
             skip = None,         # No other stop words removed
             n_breaks = 3)        # 3 breaktpoints

The figure: Figure 4. Structural break analysis - yearly Figure 4. Structural break analysis - yearly Subsetting the data to the two Covid years (2020–2021), we might observe monthly changes in public sentiment, keeping n_breaks = 3 and setting time_freq = 'M' : Figure 5. Structural break analysis - monthly Figure 5. Structural break analysis - monthly The graph is not very informative. There are 1577 rows for 24 time observations in this subset, and after cleaning the raw data, the time series is very volatile. Making conclusions using a clustering-based algorithm on such a limited volume of data is not a good idea.

Results interpretation: the structural break analysis in yearly frequency statstically confirmed what we could see from the time series of sentiment in Figure 3. Fisher-Jenks algorithm identified three structural breaks - in 2009, 2017, and 2021. We can only guess what caused the decline in 2009 and between 2016 and 2018. The 2021's drop is likely reasoned by the Covid-19 crisis.

4. Best practices for structural break analysis

Let's summarize the recommendations for the most effective use of _coffee break :

Image by author Image by author

Conclusion

A drawback of _coffee break is that currently, it only works with English texts. Due to the fact that Arabica is mainly a Pandas-based package (including Numpy vectorization in some parts), _coffee break is rather slow in evaluating large datasets. It is time-efficient in processing datasets of up to approx. 40 000 rows. Read these tutorials to find out more about n-gram and sentiment analysis and visualization of time-series text data:

_Coffee break has been developed in cooperation with Prof. Jitka Poměnková (Brno University of Technology). The complete code in this tutorial is on my GitHub. Did you like the article? You can invite me for coffee and support my writing. You can also subscribe to my email list to get notified about my new articles. Thanks! Photo by Content Pixie on Unsplash Photo by Content Pixie on Unsplash

References

[1] Hutto, C., Gilbert, E. (2014). VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proceedings of the International AAAI Conference on Web and Social Media , 8(1), 216–225. [2] Jenks, G.F. (1977). Optimal data classification for choropleth maps. Kansas. University. Dept. of Geography-Meteorology. Occasional paper no. 2.

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Petr Koráb

See all from Petr Koráb PythonSentiment AnalysisText MiningTime Series Analysis Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Automating Scientific Data Analysis Part 1:

Programming Why and How You Can Write Python Programs that Automatically Analyze Scientific Data Sets Peter Grant April 1, 20197 min read

Extracting Keyphrases from Text: RAKE and Gensim in Python

Machine Learning Using Python to Get Keyphrases from Large Text Passages Nikita Saxena September 6, 20204 min read

loc vs iloc in Pandas

Programming What's the difference between loc[]and iloc[] in Python and Pandas Introduction Indexing and slicing pandas... Giorgos Myrianthous June 4, 20213 min read

Natural Language Processing (NLP): Don't Reinvent the Wheel

NLP Tools and techniques to accelerate your Python NLP projects Abhijit Gupta August 12, 202010 min read

Fourier Transform, the Practical Python Implementation

Data Science A practical application on real-world signals Omar Alkousa February 27, 202311 min read

How My Learning Path Changed After Becoming a Data Scientist

Artificial Intelligence I keep learning but in a different way Soner Yıldırım June 11, 20215 min read

How to Create Beautiful Waffle Charts for Data Visualisation in Python

Data Science A Great Alternative to Pie Charts for Data Visualisation Andy McDonald February 8, 20237 min read

6 Dimensionality Reduction Techniques: How and When to use them?

Machine Learning Learn how and why you should use a specific dimensionality technique over another one Kevin Berlemont, PhD May 20, 20225 min read

A New Way to Predict Probability Distributions

Machine Learning Exploring multi-quantile regression with Catboost Harrison Hoffman February 14, 20239 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close