保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Beginner's Guide To Building A Singlish AI Chatbot

AI text generation is one of the most exciting fields in NLP, but also a daunting one for beginners. This post aims to speed up the... Chua Chin Hon August 5, 20208 min read Singlish phrases such as blur as a sotong can be bewildering for non-Singaporeans. Can a transformer model make sense of it? Photo: Chua Chin Hon Singlish phrases such as "blur as a sotong" can be bewildering for non-Singaporeans. Can a transformer model make sense of it? Photo: Chua Chin Hon

AI text generation is one of the most exciting fields in NLP, but also a daunting one for beginners. This post aims to speed up the learning process for newcomers by combining and adapting several existing tutorials into a practical end-to-end walkthrough with notebooks and sample data for a conversational chatbot that can be used in an interactive app.

Auto-text generation is undoubtedly one of the most exciting fields in NLP in recent years. But it's also an area that's relatively difficult for newcomers to navigate, due to the high bar for technical knowledge and resource requirements. While there's no shortage of helpful notebooks and tutorials out there, pulling the various threads together can be time consuming. To help speed up the learning process for fellow newcomers, I've put together a simple end-to-end project to create a simple AI conversational chatbot that you can run in an interactive app. I chose to frame the text generation project around a chatbot as we react more intuitively to conversations, and can easily tell whether the auto-generated text is any good. Chatbots are also ubiquitous enough that most of us would have a good sense of the expected baseline performance without having to consult a manual or an expert. If it's bad, you'll know right away without having to check a score or metric. The workflow in the project is structured with the practical questions of a beginner in mind: How to get started with data extraction and preparation, what's needed to fine tune a pre-trained transformer model, and finally how to test/use the bot in an app. Learn this step by step with the interactive AI and Data Scientist roadmap. The end product is not what one might call "polished", as the screen-grab below of one of the test-runs shows. But as a first experiment, the results are good enough (in my view) for highlighting both the possibilities and limits of AI text generation via transfer learning. Screen-grab of test run of Singlish Chatbot on Colab. Screen-grab of test run of Singlish Chatbot on Colab.

DATA, REPO AND REQUIREMENTS

If the sample conversation above looks bewildering to you, well, you've likely not been to Singapore and/or heard of "Singlish", or colloquial Singaporean English. It's a mish-mash of several languages and local slang, and can be confusing for non-Singaporeans. In other words, not a bad way to test the limits of the DialoGPT model. The data for fine tuning the model is taken from a collection of SMS messages by Singaporean students at a local university. While the written and spoken forms of "Singlish" can differ significantly, we'll set that aside for practical reasons. The dataset (raw and cleaned versions for training) and four related notebooks are available in my repo. The notebooks are split according to the following tasks:

As far as resource requirements go, you can run this project on a free Google/Colab account if you fine tune a DialoGPT-small model instead of the larger versions. If you are using a more robust dataset, perhaps fine tuning a DialoGPT-small model would be sufficient. But I found that my results from fine tuning the smaller model weren't as good, and the constant housekeeping to avoid busting the 15Gb storage limit on a free Google account was a drain on productivity. If you foresee yourself experimenting with more/larger transformer models in future, I'd recommend an upgrade to Colab Pro as well as increasing the amount of storage space on your Google account. This allowed me to iterate quickly, without having to wrestle with a physical eGPU set up at home. For brevity, I won't go into the technical details in this post. I'm still learning as I go, and there are far better articles on this topic out there. Most of the code are lifted or adapted from the work of previous authors, and they are acknowledged as such in the notebooks.

STEP 1: DATA PREPARATION

The first and biggest question for the project is whether you can find an appropriate dataset for your use case. The more specific the use case (say, a Trevor Noah chatbot), the harder it will be to find the appropriate training data. Ethical questions should be considered at the outset: Is it appropriate to model the chatbot after a real person, even if you have access to transcripts, and what is the likelihood of potential misuse? The anonymised SMS dataset used in this project is among the few "Singlish" corpuses in the public space, and is the only one I've found that's large enough for this purpose. The first half of notebook3.0 involves the steps needed to extract the SMSes from a deeply nested json file. Skip this if you are using a different dataset. The second half deals with the very specific way that the training and validation data sets have to be prepared for use in the DialoGPT model. Essentially, for every SMS you have to create x-number of previous SMSes as "context": Click here to see another example of how data is prepared in building a Rick Sanchez bot. Click here to see another example of how data is prepared in building a Rick Sanchez bot. I settled on seven previous responses, but you can change it up or down to see what impact this may or may not have on the chatbot's performance. If this is more than an experiment for you, I suspect this is where you'll be spending a lot of time tweaking the dataset to clean up the response/context. Unfortunately, I've not come across a good tutorial on how best to structure or tweak custom datasets for fine tuning a DialoGPT model. After splitting the response-context dataset into training and validation sets, you are pretty much set for the fine tuning.

STEP 2: FINE TUNING DIALOGPT

There's a tonne of code involved in the fine tuning process, but thanks to the earlier work by folks like Nathan Cooper and Rostyslav Neskorozhenyi, the training loops can be used as is without new users having to make major changes. You can study the code in greater detail when you have more time. The key cell to note in notebook3.1 is the one below: This is where you'd need to make changes depending on your dataset and the set-up at your disposal. For example, you can stick with the medium-sized DialoGPT model or dial down to the small one. If you encounter GPU-out-of-memory issues, you'll have to reduce the batch size (as I did in cell above by reducing to 1). If you don't have much storage space left in your Google Drive, you'll have to adjust the logging_steps and save_steps intervals accordingly (files saved at various checkpoints can chew up storage space in a hurry). The fine tuning process can take anything from 40 minutes to about 2 hours, depending on the parameters you set. Do note that there are usage limits even on Colab Pro. For instance, I wasn't able to fine tune a DialoGPT-large model due to GPU memory limits. Colab Pro notebooks can run up to 24 hours, but I have yet to test that out with more epochs. The one metric to take note of at the end of the fine tuning process is the perplexity score - a measure of how certain the model is in picking the next token. The lower the score, the better as it means the model is less uncertain. I got a perplexity score of 1.6134 from fine tuning the DialoGPT-medium model, a significant improvement from the scores in the ranges of 6.3 to 5.7 when using the DialoGPT-small model. It'll take several runs for you to figure out what's an acceptable score. You can start chatting with the bot at the end of the notebook (assuming everything ran correctly), but I much prefer to load the fine tuned model into an app. Thanks to Lu Xing Han @ Plotly, there's a notebook for that.

STEP 3: TEST YOUR FINE TUNED MODEL IN A DASH APP

The Plotly repo contains code for running the chatbot app on a local machine if you wish to do that, but I opted to run notebook3.2 on Colab for practical reasons (the fine tuned model is already there). The code is lifted straight from Lu Xing Han's demo, with minor changes towards the end for folks who might want to experiment further with the output from the bot: "Temperature" allows you to control the level of creativity (or craziness/brazenness) of the responses. You can reduce the length of the chatbot's "utterances" by lowering the max_length (max number of token: 1,024; default: 256 tokens). A full list of the arguments you can tweak for the chatbot conversation can be found here, under the section "Using the interaction script". Here's a screen-grab of slightly more successful conversation I managed to have with the chatbot. Highlighted cells are my entries: The fine tuned pytorch model is too big (1.44Gb) to be deployed on any free hosting account, so there's no way (for now) for you to try this particular Singlish chatbot on a web app. My take: The "Singlish" chatbot is not ready for prime-time. It can give surprisingly good responses at times, but can't sustain the effort beyond a few exchanges. As you can see in the screen-grab above, things go sideways towards the end. With a richer dataset and perhaps enough compute to train on a DialoGPT-large model, the responses might get better. But as an experiment, I'd have to say the performance is not half bad at all.

BONUS STEP: CPU ALTERATIVE TO AUTO-TEXT GENERATION

There are a number of alternatives out there if you'd rather not use Colab and/or confine the data and the fine-tuning to a local machine. I'll just highlight one Python library that I've been experimenting with - aitextgen - that provides an option for CPU-only training. Notebook3.3 outlines a simple example using the same SMS dataset in this project. I had previously tried aitextgen with other datasets involving YouTube transcripts of political speeches in Singapore. Unfortunately, I've not been able to get very satisfactory results so far. I'm probably not using it right. You might have better luck. Aitextgen can run on Colab too.

Hopefully this post and the accompanying notebooks will help you get started quickly on experiments with your own AI chatbot. What's far harder to do is figuring out how to improve its performance, or ensure that it's safe for public use. There are also related technical issues to explore, such as slimming the model down for deployment or more advanced interactive app designs that can bring out additional functionalities in the chatbot. If you have recommendations for any of the above topics, or spot mistakes in this post, ping me at: Twitter: Chua Chin Hon LinkedIn: www.linkedin.com/in/chuachinhon

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Chua Chin Hon

See all from Chua Chin Hon AIChatbotsData ScienceSingaporeTransfer Learning Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Analysing Pharmaceutical Sales Data in Python

Programming How to analyse pharmaceutical sales data using Python, Pandas, Matplotlib and regression models Przemyslaw Jarzynski September 6, 20209 min read

Building a Command-Line Quiz Application in R

Data Science Practice control flow, input handling, and functions in R by creating an interactive quiz game. Benjamin Nweke October 5, 20256 min read

Can LSTM be used to Forecast the Number of Sunspots?

Machine Learning I will be predicting the number of monthly Sunspots using time series analysis. The models that will be explored in-depth are the ARMA and... Ethan Johnson-Skinner, MSc September 9, 202110 min read

P value - Explained

Data Science What it means and how it is used. Explained with a use case. Soner Yıldırım July 8, 20207 min read

End to end Shiny app tutorial using NYC mortality data

Data Science Conceptual overview and step by step guide for beginners Emily A. Halford January 10, 202117 min read

A Journey through XGBoost: Milestone 1

Machine Learning Setting up the background Rukshan Pramoditha March 2, 20216 min read

The Intuitive Basics of Optimization

Data Science A gentle introduction to the amazing field of optimization Jarom Hulet June 26, 202412 min read

In defense of statistical modeling

Machine Learning A persistent thread of commentary says data science is overhyped. Don't believe it. Brian Kent June 10, 20218 min read

How to Combine the Forecasts of an Ensemble

Artificial Intelligence Using dynamic forecasting ensembles to cope with changes in the time series Vitor Cerqueira January 19, 20235 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close