{ "source_file": "0930", "source_platform": "Medium", "author": "Shawhin Talebi", "title_listed": "Fine-Tuning BERT for Text Classification", "keywords": "BERT,文本分类,微调", "domain": "文本处理", "content_type": "深度技术解析", "is_paid_flag": "部分免费/会员墙", "url": "https://towardsdatascience.com/fine-tuning-bert-for-text-classification-a01f89b179fc/", "publish_date": "2024-10", "corpus_stage": "预训练/SFT", "notes": "", "id": "0a4edb185fe9", "collected_at": "2026-10-01T22:24:55", "title": "Fine-Tuning BERT for Text Classification | Towards Data Science", "body_markdown": "![Revisit consent button](https://cdn-cookieyes.com/assets/images/revisit.svg)\nWe value your privacy\nWe use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking \"Accept All\", you consent to our use of cookies.\nCustomise Reject All Accept All\nCustomise Consent Preferences ![](https://cdn-cookieyes.com/assets/images/close.svg)\nWe use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.\nThe cookies that are categorised as \"Necessary\" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more\nNecessaryAlways Active\nNecessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.\n * Cookie\nBCTempID\n * Duration\n10 minutes\n * Description\nNo description available.\n\n\n * Cookie\n__cf_bm\n * Duration\n1 hour\n * Description\nThis cookie, set by Cloudflare, is used to support Cloudflare Bot Management. \n\n\n * Cookie\nAWSALBCORS\n * Duration\n7 days\n * Description\nAmazon Web Services set this cookie for load balancing.\n\n\n * Cookie\n_cfuvid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to track users across sessions to optimize user experience by maintaining session consistency and providing personalized services\n\n\n * Cookie\nli_gc\n * Duration\n6 months\n * Description\nLinkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.\n\n\n * Cookie\n__hssrc\n * Duration\nsession\n * Description\nThis cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.\n\n\n * Cookie\n__hssc\n * Duration\n1 hour\n * Description\nHubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.\n\n\n * Cookie\nwpEmojiSettingsSupports\n * Duration\nsession\n * Description\nWordPress sets this cookie when a user interacts with emojis on a WordPress site. It helps determine if the user's browser can display emojis properly.\n\n\n * Cookie\nBCSessionID\n * Duration\n1 year 1 month 4 days\n * Description\nBlueconic sets this cookie as a unique identifier for the BlueConic profile.\n\n\n * Cookie\n_octo\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nlogged_in\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\n__Secure-YEC\n * Duration\npast\n * Description\nYouTube sets this cookie to stores the user's video player preferences using embedded YouTube video\n\n\n * Cookie\n__eoi\n * Duration\n6 months\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTGCORS\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\nlogin-status-p\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nAWSALBTG\n * Duration\n7 days\n * Description\nNo description available.\n\n\n * Cookie\ncsrf_token\n * Duration\nsession\n * Description\nNo description available.\n\n\n * Cookie\ntoken_v2\n * Duration\n1 day\n * Description\nDescription is currently not available.\n\n\n * Cookie\nD\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nPHPSESSID\n * Duration\nsession\n * Description\nThis cookie is native to PHP applications. The cookie stores and identifies a user's unique session ID to manage user sessions on the website. The cookie is a session cookie and will be deleted when all the browser windows are closed.\n\n\n * Cookie\nVISITOR_PRIVACY_METADATA\n * Duration\n6 months\n * Description\nYouTube sets this cookie to store the user's cookie consent state for the current domain. \n\n\n * Cookie\ncookietest\n * Duration\nsession\n * Description\nThe cookietest cookie is typically used to determine whether the user's browser accepts cookies, essential for website functionality and user experience.\n\n\n * Cookie\n__Host-airtable-session\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\n__Host-airtable-session.sig\n * Duration\n1 year\n * Description\nThis cookie is used to enable us to integrate the services of Airtable.\n\n\n * Cookie\nm\n * Duration\n1 year 1 month 4 days\n * Description\nStripe sets this cookie for fraud prevention purposes. It identifies the device used to access the website, allowing the website to be formatted accordingly.\n\n\n * Cookie\nBIGipServer*\n * Duration\nsession\n * Description\nMarketo sets this cookie to collect information about the user's online activity and build a profile about their interests to provide advertisements relevant to the user. \n\n\n * Cookie\n__cfruid\n * Duration\nsession\n * Description\nCloudflare sets this cookie to identify trusted web traffic.\n\n\n * Cookie\n_GRECAPTCHA\n * Duration\n6 months\n * Description\nGoogle Recaptcha service sets this cookie to identify bots to protect the website against malicious spam attacks.\n\n\n * Cookie\n__Secure-YNID\n * Duration\n6 months\n * Description\nGoogle cookie used to protect user security and prevent fraud, especially during the login process.\n\n\n * Cookie\ncookieyes-consent\n * Duration\n1 year\n * Description\nCookieYes sets this cookie to remember users' consent preferences so that their preferences are respected on subsequent visits to this site. It does not collect or store any personal information about the site visitors.\n\n\nFunctional\nFunctional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.\n * Cookie\nlidc\n * Duration\n1 day\n * Description\nLinkedIn sets the lidc cookie to facilitate data center selection.\n\n\n * Cookie\nbrw\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nbrwConsent\n * Duration\n5 minutes\n * Description\nDescription is currently not available.\n\n\n * Cookie\nWMF-Uniq\n * Duration\n1 year\n * Description\nDescription is currently not available.\n\n\n * Cookie\nloom_anon_comment\n * Duration\n1 year\n * Description\nNo description available.\n\n\n * Cookie\nloom_referral_video\n * Duration\nsession\n * Description\nDescription is currently not available.\n\n\n * Cookie\nVISITOR_INFO1_LIVE\n * Duration\n6 months\n * Description\nA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.\n\n\n * Cookie\nyt-remote-connected-devices\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nytidb::LAST_RESULT_ENTRY_KEY\n * Duration\nNever Expires\n * Description\nThe cookie ytidb::LAST_RESULT_ENTRY_KEY is used by YouTube to store the last search result entry that was clicked by the user. This information is used to improve the user experience by providing more relevant search results in the future.\n\n\n * Cookie\nyt-remote-device-id\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to store the user's video preferences using embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-name\n * Duration\nsession\n * Description\nThe yt-remote-session-name cookie is used by YouTube to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\nyt-remote-fast-check-period\n * Duration\nsession\n * Description\nThe yt-remote-fast-check-period cookie is used by YouTube to store the user's video player preferences for embedded YouTube videos.\n\n\n * Cookie\nyt-remote-session-app\n * Duration\nsession\n * Description\nThe yt-remote-session-app cookie is used by YouTube to store user preferences and information about the interface of the embedded YouTube video player.\n\n\n * Cookie\nyt-remote-cast-available\n * Duration\nsession\n * Description\nThe yt-remote-cast-available cookie is used to store the user's preferences regarding whether casting is available on their YouTube video player.\n\n\n * Cookie\nyt-remote-cast-installed\n * Duration\nsession\n * Description\nThe yt-remote-cast-installed cookie is used to store the user's video player preferences using embedded YouTube video.\n\n\n * Cookie\ncp_session\n * Duration\n3 months\n * Description\nCodepen sets this cookie for Help systems found in the website.\n\n\n * Cookie\nloid\n * Duration\n1 year 1 month 4 days\n * Description\nThis cookie is set by the Reddit. The cookie enables the sharing of content from the website onto the social media platform.\n\n\nAnalytics\nAnalytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.\n * Cookie\n__hstc\n * Duration\n6 months\n * Description\nHubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).\n\n\n * Cookie\nhubspotutk\n * Duration\n6 months\n * Description\nHubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.\n\n\n * Cookie\n_ga\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.\n\n\n * Cookie\n_ga_*\n * Duration\n1 year 1 month 4 days\n * Description\nGoogle Analytics sets this cookie to store and count page views.\n\n\n * Cookie\n__Host-psifi.analyticsTrace\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n__Host-psifi.analyticsTraceV2\n * Duration\n6 hours\n * Description\nDescription is currently not available.\n\n\n * Cookie\n_gh_sess\n * Duration\nsession\n * Description\nGitHub sets this cookie for temporary application and framework state between pages like what step the user is on in a multiple step form.\n\n\n * Cookie\nYSC\n * Duration\nsession\n * Description\nYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.\n\n\n * Cookie\najs_anonymous_id\n * Duration\n1 year\n * Description\nThis cookie is set by Segment to count the number of people who visit a certain site by tracking if they have visited before.\n\n\n * Cookie\nvuid\n * Duration\n1 year 1 month 4 days\n * Description\nVimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. \n\n\nPerformance\nPerformance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.\n * Cookie\nAWSALB\n * Duration\n7 days\n * Description\nAWSALB is an application load balancer cookie set by Amazon Web Services to map the session to the target.\n\n\n * Cookie\nacq\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nacq.sig\n * Duration\npast\n * Description\nDescription is currently not available.\n\n\n * Cookie\nptc\n * Duration\n2 years\n * Description\nNo description available.\n\n\nAdvertisement\nAdvertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.\n * Cookie\nmuc_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to collect user behaviour and interaction data to optimize the website.\n\n\n * Cookie\nguest_id_marketing\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\nguest_id_ads\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor.\n\n\n * Cookie\npersonalization_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to integrate and share features for social media and also store information about how the user uses the website, for tracking and targeting.\n\n\n * Cookie\nguest_id\n * Duration\n1 year 1 month 4 days\n * Description\nTwitter sets this cookie to identify and track the website visitor. It registers if a user is signed in to the Twitter platform and collects information about ad preferences.\n\n\n * Cookie\nbcookie\n * Duration\n1 year\n * Description\nLinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.\n\n\n * Cookie\n__Secure-ROLLOUT_TOKEN\n * Duration\n6 months\n * Description\nYouTube sets this cookie to manage feature rollout and experimentation. It helps Google control which new features or interface changes are shown to users as part of testing and staged rollouts, ensuring consistent experience for a given user during an experiment.\n\n\n * Cookie\nyt.innertube::nextId\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nyt.innertube::requests\n * Duration\nNever Expires\n * Description\nYouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.\n\n\n * Cookie\nsession_tracker\n * Duration\nsession\n * Description\nThis cookie is set by the Reddit. This cookie is used to identify trusted web traffic. It also helps in adverstising on the website.\n\n\n * Cookie\nedgebucket\n * Duration\nsession\n * Description\nReddit sets this cookie to save the information about a log-on Reddit user, for the purpose of advertisement recommendations and updating the content.\n\n\n * Cookie\ndid\n * Duration\n1 year\n * Description\nArbor sets this cookie to show targeted ads to site visitors.This cookie expires after 2 months or 1 year.\n\n\nUncategorised\nOther uncategorised cookies are those that are being analysed and have not been classified into a category as yet.\nNo cookies to display.\nReject All Save My Preferences Accept All\nToggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Fine-Tuning BERT for Text Classification\nA hackable example with Python code\n[Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\nOctober 17, 20246 min read\nAlthough today's 100B+ parameter transformer models are state-of-the-art in AI, there's still much we can accomplish with smaller (< 1B parameter) models. In this article, I will walk through one such example, fine-tuning BERT (110M parameters) to classify phishing URLs. I'll start by covering key concepts and then share example Python code.\n![Image from Canva.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/1S-5K3D3dt8EH_S2eLE8aSQ.png)\nImage from Canva.\n* * *\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## **Fine-tuning**\n[**Fine-tuning**](https://towardsdatascience.com/fine-tuning-large-language-models-llms-23473d763b91) **involves** **adapting a pre-trained model to a particular use case through additional training**.\nPre-trained models are developed via unsupervised learning, which precludes the need for large-scale labeled datasets. Fine-tuned models can then exploit pre-trained model representations to significantly **reduce training costs** and **improve model performance** compared to training from scratch [1].\n> [**Fine-Tuning Large Language Models (LLMs)**](https://towardsdatascience.com/fine-tuning-large-language-models-llms-23473d763b91)\nSplitting the training process into multiple phases has led to today's state-of-the-art transformer models, such as GPT-4o, Claude, and Llama 3.2. It also **enables the democratization of AI** since the expensive undertaking of model pre-training can be done by specialized research labs, who can then make these models publicly available for fine-tuning.\n## **BERT**\nWhile model fine-tuning gained tremendous popularity post-ChatGPT, it's been around since (at least) 2015 [2]. One of the early language models developed specifically for fine-tuning was Google's BERT model, which was pre-trained on two unsupervised tasks: **1) masked language modeling (MLM)** and **2) next sentence prediction** [1].\nThe **MLM** pre-training task consists of **predicting arbitrarily masked words in a sequence**. This is in contrast to causal language modeling, which is restricted to predicting the word at the end of a sequence. Therefore, MLM enables models to leverage more context (i.e. text before AND after the masked word) to make predictions [1].\n**Next sentence prediction** is important for downstream tasks that require understanding the relationship between two sentences (e.g., Question Answering and Semantic Similarity). This is implemented using special input tokens to distinguish the sentence prediction task from the MLM [1].\nThese pre-training tasks enable BERT to be fine-tuned on a wide range of tasks such as sentiment analysis, sentence similarity, question answering, named entity recognition, common sense reasoning, and many others [1].\n## **Text Classification**\nMany of the tasks mentioned above (e.g. sentiment analysis, sentence similarity, named entity recognition) fall under the category of **text classification** , i.e., **assigning a label to input text sequences**.\nThere are countless practical applications of text classification, such as detecting spam in emails, categorizing IT support tickets, detecting toxic or harmful speech, and analyzing the sentiment of customer reviews. While each of these tasks is practically very different, their implementations are almost identical from a technical standpoint.\n## **Example Code: Fine-tuning BERT for Phishing URL Identification**\nHere, we will walk through an example of BERT fine-tuning to classify phishing URLs. We will use the [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased) model freely available on the Hugging Face (HF) hub.\nThe model consists of 110M parameters, of which we will only train a small percentage. Therefore, this **example should easily run on most consumer hardware (no GPU required)**.\nThe fine-tuned model is also available on the [HF hub](https://huggingface.co/shawhin/bert-phishing-classifier_teacher), and an example notebook is available on [GitHub](https://github.com/ShawhinT/YouTube-Blog/blob/main/LLMs/model-compression/0_train-teacher.ipynb).\n* * *\nWe'll start by importing a few handy libraries.\npython\n\n```\nfrom datasets import DatasetDict, Dataset\nfrom transformers import AutoTokenizer, AutoModelForSequenceClassification, \n TrainingArguments, Trainer\nimport evaluate\nimport numpy as np\nfrom transformers import DataCollatorWithPadding\n```\n\nNext, we'll load the training dataset. It consists of 3,000 text-label pairs with a 70–15–15 train-test-validation split. The data are originally from [here](https://www.kaggle.com/datasets/taruntiwarihp/phishing-site-urls/data) (open database license).\npython\n\n```\ndataset_dict = load_dataset(\"shawhin/phishing-site-classification\")\n```\n\nThe Transformer library makes it super easy to **load and adapt pre-trained models**. Here's what that looks like for the BERT model.\npython\n\n```\n# define pre-trained model path\nmodel_path = \"google-bert/bert-base-uncased\"\n\n# load model tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_path)\n\n# load model with binary classification head\nid2label = {0: \"Safe\", 1: \"Not Safe\"}\nlabel2id = {\"Safe\": 0, \"Not Safe\": 1}\nmodel = AutoModelForSequenceClassification.from_pretrained(model_path, \n num_labels=2, \n id2label=id2label, \n label2id=label2id,)\n```\n\nWhen we load a model like this, all the parameters will be set as trainable by default. However, training all 110M parameters will be computationally costly and potentially unnecessary.\nInstead, we can freeze most of the model parameters and **only train the model's final layer and classification head**.\ngraphql\n\n```\n# freeze all base model parameters\nfor name, param in model.base_model.named_parameters():\n param.requires_grad = False\n\n# unfreeze base model pooling layers\nfor name, param in model.base_model.named_parameters():\n if \"pooler\" in name:\n param.requires_grad = True\n```\n\nNext, we will need to **preprocess our data**. This will consist of two key operations: tokenizing the URLs (i.e., converting them into integers) and truncating them.\npython\n\n```\n# define text preprocessing\ndef preprocess_function(examples):\n # return tokenized text with truncation\n return tokenizer(examples[\"text\"], truncation=True)\n\n# preprocess all datasets\ntokenized_data = dataset_dict.map(preprocess_function, batched=True)\n```\n\nAnother important step is creating a **data collator** that will dynamically pad token sequences in a batch during training so they have the same length. We can do this in one line of code.\npython\n\n```\n# create data collator\ndata_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n```\n\nAs a final step before training, we can define a function to compute a set of metrics to help us monitor training progress. Here, we will consider model accuracy and AUC.\npython\n\n```\n# load metrics\naccuracy = evaluate.load(\"accuracy\")\nauc_score = evaluate.load(\"roc_auc\")\n\ndef compute_metrics(eval_pred):\n # get predictions\n predictions, labels = eval_pred\n\n # apply softmax to get probabilities\n probabilities = np.exp(predictions) / np.exp(predictions).sum(-1, \n keepdims=True)\n # use probabilities of the positive class for ROC AUC\n positive_class_probs = probabilities[:, 1]\n # compute auc\n auc = np.round(auc_score.compute(prediction_scores=positive_class_probs, \n references=labels)['roc_auc'],3)\n\n # predict most probable class\n predicted_classes = np.argmax(predictions, axis=1)\n # compute accuracy\n acc = np.round(accuracy.compute(predictions=predicted_classes, \n references=labels)['accuracy'],3)\n\n return {\"Accuracy\": acc, \"AUC\": auc}\n```\n\nNow, we are ready to fine-tune our model. We start by defining hyperparameters and other training arguments.\npython\n\n```\n# hyperparameters\nlr = 2e-4\nbatch_size = 8\nnum_epochs = 10\n\ntraining_args = TrainingArguments(\n output_dir=\"bert-phishing-classifier_teacher\",\n learning_rate=lr,\n per_device_train_batch_size=batch_size,\n per_device_eval_batch_size=batch_size,\n num_train_epochs=num_epochs,\n logging_strategy=\"epoch\",\n eval_strategy=\"epoch\",\n save_strategy=\"epoch\",\n load_best_model_at_end=True,\n)\n```\n\nThen, we pass our training arguments into a trainer class and train the model.\nmakefile\n\n```\ntrainer = Trainer(\n model=model,\n args=training_args,\n train_dataset=tokenized_data[\"train\"],\n eval_dataset=tokenized_data[\"test\"],\n tokenizer=tokenizer,\n data_collator=data_collator,\n compute_metrics=compute_metrics,\n)\n\ntrainer.train()\n```\n\nThe training results are shown below. We can see that the training and validation loss are monotonically decreasing while the accuracy and AUC increase with each epoch.\n![Training results. Image by author.](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/1T71pn8_rrQHGDvS3NzyNZQ.png)\nTraining results. Image by author.\nAs a final test, we can evaluate the performance of the model on the independent validation data, i.e., data not used for training or setting hyperparameters.\npython\n\n```\n# apply model to validation dataset\npredictions = trainer.predict(tokenized_data[\"validation\"])\n\n# Extract the logits and labels from the predictions object\nlogits = predictions.predictions\nlabels = predictions.label_ids\n\n# Use your compute_metrics function\nmetrics = compute_metrics((logits, labels))\nprint(metrics)\n\n# >> {'Accuracy': 0.889, 'AUC': 0.946}\n```\n\n_**Bonus**_ : Although a 110M parameter model is tiny compared to modern language models, we can reduce its computational requirements using [model compression techniques](https://towardsdatascience.com/compressing-large-language-models-llms-9f406eea5b5e). I cover how to reduce the memory footprint model by 7X in the article below.\n> [**Compressing Large Language Models (LLMs)**](https://towardsdatascience.com/compressing-large-language-models-llms-9f406eea5b5e)\n## Conclusion\nFine-tuning pre-trained models is a powerful paradigm for developing better models at a lower cost than training them from scratch. Here, we saw how to do this with BERT using the Hugging Face Transformers library.\nWhile the example code was for URL classification, it can be readily adapted to other text classification tasks.\nMore on LLMs 👇\n> [**Large Language Models (LLMs)**](https://shawhin.medium.com/list/8e009ae3054c)\n* * *\n**My website** : \n[1] [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805)\n[2] [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432)\n## Register Now to Continue Reading\n### Unlock Towards Data Science with a Free Account\nAll articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more.\nRegister for freeSign in to your account\n* * *\n![](https://assets.insightmediagroup.io/media/1787657074748_utev3i.jpeg)\nWritten By\n## [Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[See all from Shaw Talebi](https://towardsdatascience.com/author/shawhin/)\n[Bert](https://towardsdatascience.com/tag/bert/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Fine Tuning](https://towardsdatascience.com/tag/fine-tuning/)[Llm](https://towardsdatascience.com/tag/llm/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)\nShare This Article\n * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-bert-for-text-classification-a01f89b179fc%2F&title=Fine-Tuning%20BERT%20for%20Text%20Classification)\n * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-bert-for-text-classification-a01f89b179fc%2F&title=Fine-Tuning%20BERT%20for%20Text%20Classification)\n * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-bert-for-text-classification-a01f89b179fc%2F&text=Fine-Tuning%20BERT%20for%20Text%20Classification)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2018/07/1m7QAwwbKK9lIF6FHOrpyEQ.png)](https://towardsdatascience.com/how-the-good-old-sorting-algorithm-helps-a-great-machine-learning-technique-9e744020254b/)\n## [How the good old sorting algorithm helps a great machine learning technique](https://towardsdatascience.com/how-the-good-old-sorting-algorithm-helps-a-great-machine-learning-technique-9e744020254b/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nIn this article, we show that how the simple sorting algorithm is at the heart of solving an important problem in machine learning.\n[Tirthajyoti Sarkar](https://towardsdatascience.com/author/tirthajyoti/)\nJuly 18, 20187 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/1ia7OW4xhluKs_oknjeyuDw-scaled.jpeg)](https://towardsdatascience.com/k-nearest-neighbor-85bd50ea3e4d/)\n## [K-nearest neighbor](https://towardsdatascience.com/k-nearest-neighbor-85bd50ea3e4d/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nNeighbors do matter!!\n[Charanraj Shetty](https://towardsdatascience.com/author/charanrajshetty1993/)\nOctober 18, 20206 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/02/1qzscK6kiKglZYB2Hc2ia8Q.png)](https://towardsdatascience.com/amazon-data-scientist-interview-practice-problems-15b9b86e86c6/)\n## [Amazon's Data Scientist Interview Practice Problems](https://towardsdatascience.com/amazon-data-scientist-interview-practice-problems-15b9b86e86c6/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA walkthrough of some of Amazon's interview questions!\n[Terence Shin, MSc, MBA](https://towardsdatascience.com/author/terenceshin/)\nFebruary 26, 20205 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/0dyOa7IsNcWaVuH4e.jpg)](https://towardsdatascience.com/how-to-train-neural-networks-like-a-pro-1d2362768c1/)\n## [How to train Neural Networks like a pro!](https://towardsdatascience.com/how-to-train-neural-networks-like-a-pro-1d2362768c1/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nThe complete package for dealing with neural network training issues like Overfitting/Underfitting, Vanishing Gradient, Local Minima...\n[Indraneel Dutta Baruah](https://towardsdatascience.com/author/indraneeldb1993ds/)\nJuly 21, 20219 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1My3sZnT-qvMBmeB3hG6D0Q-scaled.jpeg)](https://towardsdatascience.com/do-not-use-chatgpt-only-to-summarize-text-bd2001db8ce7/)\n## [Do Not Use ChatGPT Only to \"Summarize Text\"](https://towardsdatascience.com/do-not-use-chatgpt-only-to-summarize-text-bd2001db8ce7/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nUnleash the lion.\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nMay 30, 20238 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/1OF2zR2ZqZs35G6hDW_IXuA.jpeg)](https://towardsdatascience.com/creating-a-plant-pet-toxicity-classifier-13b8ba6289e6/)\n## [Training with Controlled Randomness](https://towardsdatascience.com/creating-a-plant-pet-toxicity-classifier-13b8ba6289e6/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nCreating and comparing fast.ai learners\n[Kenichi Nakanishi](https://towardsdatascience.com/author/kenichinakanishi/)\nNovember 9, 20206 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/0iBnJzMK3X8rh5QvI-scaled.jpg)](https://towardsdatascience.com/why-data-science-may-not-be-for-you-21686a26cace/)\n## [Why Data Science May Not Be For You](https://towardsdatascience.com/why-data-science-may-not-be-for-you-21686a26cace/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nTell-tale signs why a career in data science may not suit you\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nMay 4, 20248 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2026/07/20260716localllmcostapplesilicon.jpg)](https://towardsdatascience.com/how-much-does-a-local-llm-actually-cost-to-run-i-measured-every-watt-on-apple-silicon/)\n## [How Much Does a Local LLM Actually Cost to Run? I Measured Every Watt on Apple Silicon](https://towardsdatascience.com/how-much-does-a-local-llm-actually-cost-to-run-i-measured-every-watt-on-apple-silicon/)\n[Large Language Models](https://towardsdatascience.com/category/artificial-intelligence/large-language-models/)\nFive models, sustained generation, real wall-socket energy at $0.31/kWh — and the surprise the RTX-3090 numbers predicted, only bigger.\n[Justin Stewart](https://towardsdatascience.com/author/justin.stewart/)\nJuly 28, 202610 min read\n * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1vvUmbHcG8Hae9lod7gatmA.jpg)](https://towardsdatascience.com/unlocking-the-power-of-machine-learning-in-analytics-practical-use-cases-and-skills-5201cf457360/)\n## [Unlocking the Power of Machine Learning in Analytics: Practical Use Cases and Skills](https://towardsdatascience.com/unlocking-the-power-of-machine-learning-in-analytics-practical-use-cases-and-skills-5201cf457360/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nYour essential machine learning checklist to excel as a data scientist in analytics\n[Yu Dong](https://towardsdatascience.com/author/ydong029/)\nJanuary 15, 20256 min read\n\n\n * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n * [X](https://x.com/TDataScience)\n * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n * [Threads](https://www.threads.net/@towardsdatascience)\n * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n#### Recommended Articles \nClose\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607631/image?etag=1781543525510&width=500&height=500) Critical Mistakes Companies Make When Integrating AI/ML into Their Processes ](https://towardsdatascience.com/critical-mistakes-companies-make-when-integrating-ai-ml-into-their-processes/ \"Critical Mistakes Companies Make When Integrating AI/ML into Their Processes\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607686/image?etag=1782161464448&width=500&height=500) How Relevance Models Foreshadowed Transformers for NLP ](https://towardsdatascience.com/how-relevance-models-foreshadowed-transformers-for-nlp/ \"How Relevance Models Foreshadowed Transformers for NLP\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607639/image?etag=1781250125848&width=500&height=500) Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1 ](https://towardsdatascience.com/javascript-fatigue-you-dont-need-js-to-build-chatgpt/ \"Javascript Fatigue: HTMX Is All You Need to Build ChatGPT — Part 1\")\n * [ ![](https://h030.towardsdatascience.com/rest/v2/contentStores/d7d3a4ab-7caf-4383-bbd2-fbf7fb4f1787/items/607930/image?etag=1782124566054&width=500&height=500) The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs ](https://towardsdatascience.com/subset-sum-problem-solved-in-linear-time-for-dense-enough-inputs/ \"The Subset Sum Problem Solved in Linear Time for Dense Enough Inputs\")\n\n\n", "metadata": { "title": "Fine-Tuning BERT for Text Classification | Towards Data Science", "description": "A hackable example with Python code", "keywords": null, "author": "Shaw Talebi", "og:locale": "en_US", "og:type": "article", "og:title": "Fine-Tuning BERT for Text Classification", "og:description": "A hackable example with Python code", "og:url": "https://towardsdatascience.com/fine-tuning-bert-for-text-classification-a01f89b179fc/", "og:site_name": "Towards Data Science", "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/1S-5K3D3dt8EH_S2eLE8aSQ.png", "og:image:width": "1280", "og:image:height": "720", "og:image:type": "image/png", "twitter:card": "summary_large_image", "twitter:site": "@TDataScience", "twitter:creator": "@TDataScience", "twitter:label1": "Written by", "twitter:data1": "Shaw Talebi", "twitter:label2": "Est. reading time", "twitter:data2": "6 min read", "article:published_time": "2024-10-17T04:31:47.000Z", "article:modified_time": "2024-10-17T04:31:47.000Z", "article:author": "https://towardsdatascience.com/author/shawhin/" }, "paywall": { "listed_paid": true, "detected": false, "marker": "", "public_only": true, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_000.txt", "chars": 255 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_001.txt", "chars": 68 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_002.txt", "chars": 581 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_003.txt", "chars": 277 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_004.txt", "chars": 259 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_005.txt", "chars": 84 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_006.txt", "chars": 938 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_007.txt", "chars": 407 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_008.txt", "chars": 258 }, { "language": "txt", "file": "attachments/0a4edb185fe9/code/block_009.txt", "chars": 357 } ], "images": [ { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/1S-5K3D3dt8EH_S2eLE8aSQ.png", "alt": "Image from Canva.", "local_path": "attachments/0a4edb185fe9/images/000.png", "bytes": 375688, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/1T71pn8_rrQHGDvS3NzyNZQ.png", "alt": "Training results. Image by author.", "local_path": "attachments/0a4edb185fe9/images/001.png", "bytes": 72660, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2018/07/1m7QAwwbKK9lIF6FHOrpyEQ.png", "alt": "", "local_path": "attachments/0a4edb185fe9/images/002.png", "bytes": 66530, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/10/1ia7OW4xhluKs_oknjeyuDw-scaled.jpeg", "alt": "", "local_path": "attachments/0a4edb185fe9/images/003.jpeg", "bytes": 686887, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/02/1qzscK6kiKglZYB2Hc2ia8Q.png", "alt": "", "local_path": "attachments/0a4edb185fe9/images/004.png", "bytes": 11554252, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/07/0dyOa7IsNcWaVuH4e.jpg", "alt": "", "local_path": "attachments/0a4edb185fe9/images/005.jpg", "bytes": 61476, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/05/1My3sZnT-qvMBmeB3hG6D0Q-scaled.jpeg", "alt": "", "local_path": "attachments/0a4edb185fe9/images/006.jpeg", "bytes": 614019, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2020/11/1OF2zR2ZqZs35G6hDW_IXuA.jpeg", "alt": "", "local_path": "attachments/0a4edb185fe9/images/007.jpeg", "bytes": 362311, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/0iBnJzMK3X8rh5QvI-scaled.jpg", "alt": "", "local_path": "attachments/0a4edb185fe9/images/008.jpg", "bytes": 185119, "downloaded": true }, { "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2025/01/1vvUmbHcG8Hae9lod7gatmA.jpg", "alt": "", "local_path": "attachments/0a4edb185fe9/images/009.jpg", "bytes": 305809, "downloaded": true } ], "audio": [], "videos": [ { "src": "https://www.youtube.com/c/TowardsDataScience", "quality": "unavailable", "local_path": "", "note": "无1080p且无720p，未交付更低清晰度" } ], "links": [ { "type": "internal", "href": "https://towardsdatascience.com/" }, { "type": "internal", "href": "https://towardsdatascience.com/latest/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/editors-pick/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/the-variable/" }, { "type": "internal", "href": "https://towardsdatascience.com/shipai/" }, { "type": "internal", "href": "https://towardsdatascience.com/submissions/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/shawhin/" }, { "type": "internal", "href": "https://towardsdatascience.com/fine-tuning-large-language-models-llms-23473d763b91" }, { "type": "internal", "href": "https://towardsdatascience.com/compressing-large-language-models-llms-9f406eea5b5e" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/bert/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/deep-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/fine-tuning/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/llm/" }, { "type": "internal", "href": "https://towardsdatascience.com/tag/machine-learning/" }, { "type": "internal", "href": "https://towardsdatascience.com/how-the-good-old-sorting-algorithm-helps-a-great-machine-learning-technique-9e744020254b/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/" }, { "type": "internal", "href": "https://towardsdatascience.com/author/tirthajyoti/" }, { "type": "internal", "href": "https://towardsdatascience.com/k-nearest-neighbor-85bd50ea3e4d/" }, { "type": "internal", "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/" }, { "type": "external", "href": "https://write.insightmediagroup.io" }, { "type": "external", "href": "https://www.linkedin.com/company/towards-data-science/" }, { "type": "external", "href": "https://x.com/TDataScience" }, { "type": "external", "href": "https://roadmap.sh/machine-learning" }, { "type": "external", "href": "https://huggingface.co/google-bert/bert-base-uncased" }, { "type": "external", "href": "https://huggingface.co/shawhin/bert-phishing-classifier_teacher" }, { "type": "external", "href": "https://github.com/ShawhinT/YouTube-Blog/blob/main/LLMs/model-compression/0_train-teacher.ipynb" }, { "type": "external", "href": "https://www.kaggle.com/datasets/taruntiwarihp/phishing-site-urls/data" }, { "type": "external", "href": "https://shawhin.medium.com/list/8e009ae3054c" }, { "type": "external", "href": "https://www.shawhintalebi.com/" }, { "type": "external", "href": "https://arxiv.org/abs/1810.04805" }, { "type": "external", "href": "https://arxiv.org/abs/1511.01432" }, { "type": "external", "href": "https://www.facebook.com/sharer/sharer.php?title=Fine-Tuning+BERT+for+Text+Classification&u=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-bert-for-text-classification-a01f89b179fc%2F" }, { "type": "external", "href": "https://www.linkedin.com/shareArticle?mini=true&title=Fine-Tuning+BERT+for+Text+Classification&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-bert-for-text-classification-a01f89b179fc%2F" }, { "type": "external", "href": "https://x.com/share?text=Fine-Tuning+BERT+for+Text+Classification&url=https%3A%2F%2Ftowardsdatascience.com%2Ffine-tuning-bert-for-text-classification-a01f89b179fc%2F" }, { "type": "external", "href": "https://www.youtube.com/c/TowardsDataScience" }, { "type": "external", "href": "https://www.threads.net/@towardsdatascience" }, { "type": "external", "href": "https://bsky.app/profile/towardsdatascience.com" }, { "type": "external", "href": "https://thenewstack.io" }, { "type": "external", "href": "https://roadmap.sh" } ], "repair_version": 2, "repaired_at": "2026-10-14T11:11:49Z", "source_url": "https://towardsdatascience.com/fine-tuning-bert-for-text-classification-a01f89b179fc/", "delivery_status": "excluded_paid", "discovery_status": "excluded_paid", "missing_reason": "按原约定排除付费/会员全文" }