保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Machine Learning

Choose the Right One: Evaluating Topic Models for Business Intelligence

Python tutorial for evaluating top-notch bigram topic models in customer email classification Petr Koráb April 24, 202510 min read Source: Freepic Topic models are used in businesses to classify brand-related text datasets (such as product and site reviews, surveys, and social media comments) and to track how customer satisfaction metrics change over time. There is a myriad of recent topic models one can choose from: the widely used BERTopic by Maarten Grootendorst (2022), the recent FASTopic presented at last year’s NeurIPS, (Xiaobao Wu et al.,2024), the Dynamic Topic Modelby Blei and Lafferty (2006), or a fresh semi-supervised Seeded Poisson Factorization model (Prostmaier et al., 2025). For a business use case, training topic models on customer texts, we often get results that are not identical and sometimes even conflicting. In business, imperfections cost money, so the engineers should place into production the model that provides the best solution and solves the problem most effectively. At the same pace that new topic models appear on the market, methods for evaluating their quality using new metrics also evolve. This practical tutorial will focus on bigram topic models , which provide more relevant information and identify better key qualities and problems for business decisions than single-word models (“delivery” vs. “poor delivery”, “stomach” vs. “sensitive stomach”, etc.). On one side, bigram models are more detailed; on the other, many evaluation metrics were not originally designed for their evaluation. To provide more background in this area, we will explore in detail:

Our example use case will show how bigram topic models (BERTopic and FASTopic) help prioritize email communication with customers on certain topics and reduce response times. Learn this step by step with the interactive Machine Learning roadmap.

1. What are topic model quality indicators?

The evaluation task should target the ideal state:

The ideal topic model should produce topics where words or bigrams (two consecutive words) in each topic are highly semantically related and distinct for each topic. In practice, this means that the words predicted for each topic are semantically similar to human judgment, and there is low duplication of words between topics. It is standard to calculate a set of metrics for each trained model to make a qualified decision on which model to place into production or use for a business decision, comparing the model performance metrics.

Bigram topic models work well with these metrics:

The reference corpus can be either internal (the training set) or external (e.g., an external email dataset). A large, external, and comparable corpus is a better choice because it can help reduce bias in training sets. Because this metric works with word frequencies,the training set and the reference corpus should be preprocessed the same way(i.e., if we remove numbers and stopwords in the training set, we should also do it in the reference corpus). The aggregate model score is the average of words across topics.

Let’s say we have the Top 4 words for one topic: “apple”, “banana”, “juice”, “smoothie” predicted by a topic model.Then SC looks at all combinations of words in the training set going from left to right, starting with the first word {apple, banana} , {apple, juice} , {apple, smoothie} then the second word {banana, juice} , {banana, smoothie} , then last word {juice, smoothie} and it counts the number of documents that contain both words, divided by the frequency of documents that contain the first word. Overall SC score for a model is the mean of all topic-level scores. Image 1. Semantic coherence by Mimno et al. (2011) illustration. Image by author. PUV (Percentage of Unique Words) calculates the share of unique words across topics in the model. PUV = 1 means that each topic in the model contains unique bigrams. Values close to 1 indicate a well-shaped, high-quality model with small word overlap between topics. [3].

The closer to 0 the SC and NIMP scores are , the more coherent the model is (bigrams predicted by the topic model for each topic are semantically similar). The closer to 1 PUV is , the easier the model is to interpret and use, because bigrams between topics do not overlap.

2. How can we prioritize email communication with topic models?

A large share of customer communication, not only in e-commerce businesses, is now solved with chatbots and personal client sections. Yet, it is common to communicate with customers by email. Many email providers offer developers broad flexibility in APIs to customize their email platform (e.g., MailChimp ,SendGrid ,Brevo). In this place, topic models make mailing more flexible and effective. In this use case, the pipeline takes the input from the incoming emails and uses the trained topic classifier to categorize the incoming email content. The outcome is the classified topic that the Customer Care (CC) Department sees next to each email. The main objective is to allow the CC staff to prioritize the categories of emails and reduce the response time to the most sensitive requests (that directly affect margin-related KPIs or OKRs). Image 2. Topic model pipeline illustration. Image by author.

3. Data and model set-ups

We will train FASTopic a nd BERTopic to classify emails into 8 and 10 topics and evaluate the quality of all model specifications. Read my previous TDS tutorial on topic modeling with these cutting-edge topic models. As a training set, we use a synthetically generated Customer Care Email dataset available on Kaggle with a GPL-3 license. The prefiltered data covers 692 incoming emails and looks like this: Image 3. Customer Care Email dataset. Image by author.

3.1. Data preprocessing

Cleaning text in the right order is essential for topic models to work in practice because it minimizes the bias of each cleaning operation. Numbers are typically removed first, followed by emojis, unless we don’t need them for special situations, such as extracting sentiment. Stopwordsfor one or more languages are removed afterward, followed by punctuation so that stopwords don’t break up into two tokens (“we’ve” -> “we” + ‘ve”). Additional tokens(company and people’s names, etc.) are removed in the next step in the clean data before lemmatization, which unifies tokens with the same semantics. Image 4. General preprocessing steps for topic modeling. Image by author

“Delivery” and “deliveries”, “box” and “Boxes”, or “Price” and “prices” share the same word root, but without lemmatization, topic models would model them as separate factors. That’s why customer emails should be lemmatized in the last step of preprocessing. Text preprocessing is model-specific:

3.2. Model compilation and training

You can check the full codes for FASTopic and BERTopic’s training with bigram preprocessing and cleaning in this repo. My previous TDS tutorials (4) and (5) explain all steps in detail. We train both models to classify 8 topics in customer email data. A simple inspection of the topic distribution shows that incoming emails to FASTopic are quite well distributed across topics. BERTopic classifies emails unevenly, keeping outliers (uncategorized tokens) in T-1 and a large share of incoming emails in T0. Image 5: Topic distribution, email classification. Image by author. Here are the predicted bigrams for both models with topic labels: Image 6: Models’ predictions. Image by author. Because the email corpus is a synthetic LLM-generated dataset, the naive labelling of the topics for both models shows topics that are:

For business purposes, topics should be labelled by the company’s insiders who know the customer base and the business priorities.

4. Model evaluation

If three out of eight classified topics are labeled differently, then which model should be deployed? Let's now evaluate the coherence and diversity for the trained BERTopic and FASTopic T-8 models.

4.1. NPMI

We need a reference corpus to calculate an NPMI for each model. The Customer IT Support Ticket Dataset from Kaggle, distributed with Attribution 4.0 International license, provides comparable data to our training set. The data is filtered to 11923 English email bodies.

  1. Calculate an NPMI for each bigram in the reference corpus with this code .
  2. Merge bigrams predicted by FASTopic and BERTopic with their NPMI scores from the reference corpus. The fewer NaNs are in the table, the more accurate the metric is.

Image 7: NPMI coherence evaluation.Image by author. 3. Average NPMIs within and across topics to get a single score for each model.

4.2. SC

With SC , we learn the context and semantic similarity of bigrams predicted by a topic model by calculating their position in the corpus in relation to other tokens. To do so, we:

  1. Create a document-term matrix (DTM) with a count of how many times each bigram appears in each document.
  2. Calculate topic SC scores by searching for bigram co-occurrences in the DTM and the bigrams predicted by topic models.
  3. Average topic SC to a model SC score.

4.3. PUV

Topic diversity PUV metric checks the duplicates of bigrams between topics in a model.

  1. Join bigrams into tokens by replacing spaces with underscores in the FASTopic and BERTopic tables of predicted bigrams.

Image 8: Topic diversity illustration. Image by author. 2. Calculate topic diversity as count of distinct tokens/ count of tokens in the tables for both models.

4.4. Model comparison

Let’s now summarize the coherence and diversity evaluation in Image 9. BERTopic models are more coherent but less diverse than FASTopic. The differences are not very large, but BERTopic suffers from uneven distribution of incoming emails into the pipeline (see charts in Image 5). Around 32% of classified emails fall into T0 , and 15% into T-1 , which covers the unclassified outliers. The models are trained with a min. of 20 tokens per topic. Increasing this parameter causes the model to be unable to train, probably because of the small data size. For this reason, FASTopic is a better choice for topic modelling in email classification with small training datasets. Image 9: Topic model evaluation metrics. Image by author. The last step is to deploy the model with topic labels in the email platform to classify incoming emails: Image 10. Topic model classification pipeline, output. Image by author.

Summary

Coherence and diversity metrics compare models with similar training setups, the same dataset, and cleaning strategy. We cannot compare their absolute values with the results of different training sessions. But they help us decide on the best model for our specific use case. They offer a relative comparison of various model specifications and help decide which model should be deployed in the pipeline. Topic models evaluation should always be the last step before model deployment in business practice. How does customer care benefit from the topic modelling exercise? After the topic model is put into production, the pipeline sends a classified topic for each email to the email platform that Customer Care uses for communicating with customers. With a limited staff, it is now possible to prioritize and respond faster to the most sensitive business requests (such as “time delays ” and “latency issues”), and change priorities dynamically. Data and complete codes for this tutorial are here. ··· Petr Korab is a Python Engineer and Founder of Text Mining Stories with over eight years of experience in Business Intelligence and NLP. Acknowledgments : I thank Tomáš Horský (Lentiamo, Prague), Martin Feldkircher, and Viktoriya Teliha (Vienna School of International Studies) for useful comments and suggestions.

References

[1] Blei, D. M., Lafferty, J. D. 2006. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning (pp. 113–120). [2] Dieng A.B., Ruiz F. J. R., and Blei D. M. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453. [3] Grootendorst, M. 2022. Bertopic: Neural Topic Modeling With A Class-Based TF-IDF Procedure. Computer Science. [4] Korab, P. Topic Modelling in Business Intelligence: FASTopic and BERTopic in Code. Towards Data Science. 22.1.2025. Accessible from: link. [5] Korab, P. Topic Modelling with BERTtopic in Python. Towards Data Science. 4.1.2024. Accessible from: link. [6] Wu, X, Nguyen, T., Ce Zhang, D., Yang Wang, W., Luu, A. T. 2024. FASTopic: A Fast, Adaptive, Stable, and Transferable Topic Modeling Paradigm. arXiv preprint: 2405.17978. [7] Mimno, D., Wallach, H., M., Talley, E., Leenders, M, McCallum. A. 2011. Optimizing Semantic Coherence in Topic Models. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing. [8] Prostmaier, B., Vávra, J., Grün, B., Hofmarcher., P. 2025. Seeded Poisson Factorization: Leveraging domain knowledge to fit topic models. arXiv preprint: 2405.17978.

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Petr Koráb

See all from Petr Koráb Editor's PicksBertopicModel EvaluationText ClassificationTopic Modeling Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

How to Learn Complex Topics the Right Way

Machine Learning Our weekly selection of must-read Editors' Picks and original features TDS Editors April 7, 20223 min read

Neural Graph Databases

Artificial Intelligence A new milestone in graph data management Michael Galkin March 28, 202315 min read

Analytics Frameworks Every Data Scientist Should Know

Data Science Why I believe my experience at McKinsey made me a better data scientist Tessa Xie August 27, 202410 min read

In defense of weight-sharing for neural architecture search: an optimization perspective

Artificial Intelligence How to optimize in the right geometry for provably better, faster neural architecture search. Liam Li July 16, 202011 min read

Data quality dimensions in IBM Watson Knowledge Catalog

Data Science Understand the standard data quality dimensions used by IBM Cloud Pak for Data and IBM Infosphere Information Server. Yannick Saillet November 13, 202015 min read

Using SimpleElastix to compare pre- and post-expansion microscopy images

Conventional optical microscopy is resolution-limited to about 200nm depending on the wavelength of the used light. Smaller structures can... Dr. Sebastian Reinhard October 11, 20214 min read

Why AI Is NOT Stealing Your Job

Artificial Intelligence AI does not decide who gets fired. Companies do. Marco Baity-Jesi June 3, 20267 min read

Understanding Kolmogorov-Smirnov (KS) Tests for Data Drift on Profiled Data

Machine Learning Data drift meets data profiling Felipe Adachi December 14, 202211 min read

Ensembles: the almost free Lunch in Machine Learning

Deep Learning Build optimal ensembles of neural networks with PyTorch and NumPy Dmitry Borisenko August 14, 202018 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close