保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Artificial Intelligence

What to Study if you Want to Master LLMs

What foundational concepts should you study if you want to understand Large Language Models? Ivo Bernardo August 12, 20245 min read Image by solenfeyissa @ Unsplash.com Image by solenfeyissa @ Unsplash.com Most of the code we use to interact with LLMs (Large Language Models) is hidden behind several APIs - and that's a good thing. But if you are like me, and want to understand the ins and outs of these magical models, there's still hope for you. Currently, apart from the researchers working on developing and training new LLMs, there's mostly two types of people playing with these types of models:

The problem is - and you may have felt it - that there is a fundamental knowledge in text mining and natural language processing that is completely hidden away in consumer products or APIs. And don't take me wrong - they are great for developing cool use cases around these technologies. But, if you want to a have deeper knowledge to build complex use cases or manipulate LLMs a bit better, you'll need to check the fundamentals - particularly when the models behave as you aren't expecting. In this blog post, I'm going to detail some of the most fundamental concepts you should check, if you want to be able to master large language models on a technical level! Let's start!

Learn this step by step with the interactive AI Engineer and Prompt Engineering roadmaps.

Basic NLP /NLTK

Basic natural language processing (NLP) is the first concept you should learn. Working with the traditional NLP pipelines will be a great lesson on how computers struggle to naturally understanding written text. With NLTK (Natural Language Toolkit), you can get the first contact with manipulating text within the context of machine learning. Poking around the NLTK library is awesome. This is one of first Python libraries to specialize in text mining within the open source world. It contains the most basic techniques to develop some nice prototypes, such as tokenization, stemming, lemmatization, part-of-speech tagging or named entity recognition. With its extensive documentation and community, NLTK is a great way to start getting your hands dirty with Natural Language Processing.

Word2Vec

By working with NLTK, you will understand that advanced AI use cases are impossible to build just using classical machine learning. While you can create basic sentiment analysis or text generation pipelines, the performance of these systems will fall significantly short as you add more complexity. So.. how did we get here? To the stage where we can have generalist tools that can crush the Turing Test? One of the most famous papers that revolutionized the NLP industry was the word2vec paper. Although some research was already rolling before it, this paper brough Word Vectors to mainstream attention. It was an absolute game changer. After word2vec, humans found a way to represent words mathematically, while keeping two important features:

How are these vectors built? Mostly, by training a neural network to predict a word within a context. The weights mapped on this neural network for a specific word translate into a mathematical relationship similar to the one below: Similarity between Vectors in Space - Image source: https://en.wikipedia.org/wiki/Word2vec Similarity between Vectors in Space - Image source: https://en.wikipedia.org/wiki/Word2vec Note: 2 dimensional simplification of word vectors The fact that we can build mathematical representation of words that keep the semantic relationship of language was a much needed breakthrough in the NLP field. Word vectors are a method of word embeddings, one of the building blocks of Large Language Models.

Text Classification

Next stop is combining both embeddings and simple machine learning pipelines to understand how we can transform text into features to use in machine learning models. In text classification, you normally start with logistic regression, naive bayes classifiers or tree-based models. This is where you should experiment with different tokenizers, pre-processing methods and embeddings - and where you will see performance increasing or decreasing based on your choices. Some of the most common text classification projects you can experiment are:

Check out some of the following competitions on kaggle:

Text Generation

Another area to explore is text generation. This is a very important part of Large Language Models, particularly as most of the applications that we some of level of next-word prediction. When it comes to text generation, there are two main routes to study:

Markov chains are a great way to learn about the intution of text generation. Although they tend to simulate text that leans on repetitive patterns, they are a great way to start. Naturally, as time goes by, you'll want to learn more about recurrent neural networks / embedding based methods that will completely improve the coherence and quality of your generated text. Markov Chain Example - Image Source: https://en.wikipedia.org/wiki/Markov_chain#/media/File:Markovkate_01.svg Markov Chain Example - Image Source: https://en.wikipedia.org/wiki/Markov_chain#/media/File:Markovkate_01.svg

Attention and Transformers

Finally, only after studying the foundational concepts I've laid out here, should you tackle the Attention mechanism. The Attention paper was out in 2017 and it completely transformed the NLP field. If it wasn't for the attention mechanism, the applications we see today would be completely impossible. The attention mechanism also relies on a deep knowledge on neural networks so, while you study them, it will be very helpful to understand how this mechanism fits in on the overall neural network theory. Based on the attention mechanism, the Transformer was born. It dethroned recurrent neural networks as the standard for text generation and understanding and helped to consolidate the attention mechanism as the standard for NLP applications. Learning attention and transformers may be challenging, but it will be much easier if you are armed with the basic NLP knowledge first :-)

Thank you for taking the time to read this post! In summary, here are the recommendations of a study plan to understand how Large Language Models work:

I've set up a Udemy course with these foundational methods here: Natural Language Processing Course - Image by Author Natural Language Processing Course - Image by Author

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Ivo Bernardo

See all from Ivo Bernardo Artificial IntelligenceChatGPTLlmNaturallanguageprocessingText Mining Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Your AI Is Only As Good As Your Data Quality!

Artificial Intelligence The importance of data preparation in the process of model creation A common misconception among... Praneeth Vasarla May 15, 20219 min read

5 Concrete Benefits of Bayesian Statistics

Artificial Intelligence Understand the benefits before diving into the math Renato Boemer October 24, 20215 min read

Intuition and Implementation of Non-Max Suppression Algorithm in Object Detection

Artificial Intelligence This article explains the usage and implementation of the Non-Max Suppression technique used for object detection in an image Prateek Chhikara February 1, 20225 min read

Comparing sample distributions with the Kolmogorov-Smirnov (KS) test

Artificial Intelligence How to compare samples and understand if they come from the same distribution using python Vinicius Trevisan February 7, 20226 min read

Sentiment Analysis : Simplified

Machine Learning Take a peek into the 'Hello World' of Natural Language Processing Prakhar Ganesh June 30, 20193 min read

Gaussian Process Regression From First Principles

Artificial Intelligence Gaussian Process Regression is a remarkably powerful class of machine learning algorithms. Here, we introduce them from first principles. Ryan S March 15, 202115 min read

Don't Fear Artificial General Intelligence

Artificial Intelligence People worry that AI will progress to the point where evil robots gain artificial general intelligence and take over the world. Don't... Steve Shwartz November 13, 202010 min read

3 Silent Pandas Mistakes You Should Be Aware Of

Artificial Intelligence And how they can cause hidden failures Soner Yıldırım August 15, 20235 min read

The Machine Learning “Advent Calendar” Day 20: Gradient Boosted Linear Regression in Excel

Machine Learning From random ensembles to optimization: gradient boosting explained Angela and Kezhan Shi December 22, 20259 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network