{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Jay Alammar",
  "title_listed": "A Visual Guide to Using BERT for the First Time",
  "keywords": "BERT,微调,实战",
  "domain": "文本处理",
  "content_type": "图解教程",
  "is_paid_flag": "否",
  "url": "https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/",
  "publish_date": "2019-11",
  "corpus_stage": "预训练/SFT",
  "notes": "BERT首次使用可视化指南",
  "id": "2b93130a1e7d",
  "collected_at": "2026-10-02T14:18:45",
  "title": "A Visual Guide to Using BERT for the First Time – Jay Alammar – Visualizing machine learning one concept at a time.",
  "body_markdown": "A Visual Guide to Using BERT for the First Time\nTranslations:\nChinese\n,\nKorean\n,\nRussian\nProgress has been rapidly accelerating in machine learning models that process language over the last couple of years. This progress has left the research lab and started powering some of the leading digital products. A great example of this is the\nrecent announcement of how the BERT model is now a major force behind Google Search\n. Google believes this step (or progress in natural language understanding as applied in search) represents “the biggest leap forward in the past five years, and one of the biggest leaps forward in the history of Search”.\nThis post is a simple tutorial for how to use a variant of BERT to classify sentences. This is an example that is basic enough as a first intro, yet advanced enough to showcase some of the key concepts involved.\nAlongside this post, I’ve prepared a notebook. You can see it here\nthe notebook\nor\nrun it on colab\n.\nDataset: SST2\nThe dataset we will use in this example is\nSST2\n, which contains sentences from movie reviews, each labeled as either positive (has the value 1) or negative (has the value 0):\nsentence\nlabel\na stirring , funny and finally transporting re imagining of beauty and the beast and 1930s horror films\n1\napparently reassembled from the cutting room floor of any given daytime soap\n0\nthey presume their audience won't sit still for a sociology lesson\n0\nthis is a visually stunning rumination on love , memory , history and the war between art and commerce\n1\njonathan parker 's bartleby should have been the be all end all of the modern office anomie films\n1\nModels: Sentence Sentiment Classification\nOur goal is to create a model that takes a sentence (just like the ones in our dataset) and produces either 1 (indicating the sentence carries a positive sentiment) or a 0 (indicating the sentence carries a negative sentiment). We can think of it as looking like this:\nUnder the hood, the model is actually made up of two model.\nDistilBERT\nprocesses the sentence and passes along some information it extracted from it on to the next model. DistilBERT is a smaller version of BERT developed and open sourced by the team at\nHuggingFace\n. It’s a lighter and faster version of BERT that roughly matches its performance.\nThe next model, a basic Logistic Regression model from scikit learn will take in the result of DistilBERT’s processing, and classify the  sentence as either positive or negative (1 or 0, respectively).\nThe data we pass between the two models is a vector of size 768. We can think of this of vector as an embedding for the sentence that we can use for classification.\nIf you’ve read my previous post,\nIllustrated BERT\n, this vector is the result of the first position (which receives the [CLS] token as input).\nModel Training\nWhile we’ll be using two models, we will only train the logistic regression model. For DistillBERT, we’ll use a model that’s already pre-trained and has a grasp on the English language. This model, however is neither trained not fine-tuned to do sentence classification. We get some sentence classification capability, however, from the general objectives BERT is trained on. This is especially the case with BERT’s output for the first position (associated with the [CLS] token). I believe that’s due to BERT’s second training object – Next sentence classification. That objective seemingly trains the model to encapsulate a sentence-wide sense to the output at the first position. The\ntransformers\nlibrary provides us with an implementation of DistilBERT as well as pretrained versions of the model.\nTutorial Overview\nSo here’s the game plan with this tutorial. We will first use the trained distilBERT to generate sentence embeddings for 2,000 sentences.\nWe will not touch distilBERT after this step. It’s all Scikit Learn from here. We do the usual train/test split on this dataset:\nTrain/test split for the output of distilBert (model #1) creates the dataset we'll train and evaluate logistic regression on (model #2). Note that in reality, sklearn's train/test split shuffles the examples before making the split, it doesn't just take the first 75% of examples as they appear in the dataset.\nThen we train the logistic regression model on the training set:\nHow a single prediction is calculated\nBefore we dig into the code and explain how to train the model, let’s look at how a trained model calculates its prediction.\nLet’s try to classify the sentence “a visually stunning rumination on love”. The first step is to use the BERT tokenizer to first split the word into tokens. Then, we add the special tokens needed for sentence classifications (these are [CLS] at the first position, and [SEP] at the end of the sentence).\nThe third step the tokenizer does is to replace each token with its id from the embedding table which is a component we get with the trained model. Read\nThe Illustrated Word2vec\nfor a background on word embeddings.\nNote that the tokenizer does all these steps in a single line of code:\ntokenizer\n.\nencode\n(\n\"a visually stunning rumination on love\"\n,\nadd_special_tokens\n=\nTrue\n)\nOur input sentence is now the proper shape to be passed to DistilBERT.\nIf you’ve read\nIllustrated BERT\n, this step can also be visualized in this manner:\nFlowing Through DistilBERT\nPassing the input vector through DistilBERT works\njust like BERT\n. The output would be a vector for each input token. each vector is made up of 768 numbers (floats).\nBecause this is a sentence classification task, we ignore all except the first vector (the one associated with the [CLS] token). The one vector we pass as the input to the logistic regression model.\nFrom here, it’s the logistic regression model’s job to classify this vector based on what it learned from its training phase. We can think of a prediction calculation as looking like this:\nThe training is what we’ll discuss in the next section, along with the code of the entire process.\nThe Code\nIn this section we’ll highlight the code to train this sentence classification model. A notebook containing all this code is available on\ncolab\nand\ngithub\n.\nLet’s start by importing the tools of the trade\nimport\nnumpy\nas\nnp\nimport\npandas\nas\npd\nimport\ntorch\nimport\ntransformers\nas\nppb\n# pytorch transformers\nfrom\nsklearn.linear_model\nimport\nLogisticRegression\nfrom\nsklearn.model_selection\nimport\ncross_val_score\nfrom\nsklearn.model_selection\nimport\ntrain_test_split\nThe dataset is\navailable\nas a file on github, so we just import it directly into a pandas dataframe\ndf\n=\npd\n.\nread_csv\n(\n'https://github.com/clairett/pytorch-sentiment-classification/raw/master/data/SST2/train.tsv'\n,\ndelimiter\n=\n'\n\\t\n'\n,\nheader\n=\nNone\n)\nWe can use df.head() to look at the first five rows of the dataframe to see how the data looks.\ndf\n.\nhead\n()\nWhich outputs:\nImporting pre-trained DistilBERT model and tokenizer\nmodel_class\n,\ntokenizer_class\n,\npretrained_weights\n=\n(\nppb\n.\nDistilBertModel\n,\nppb\n.\nDistilBertTokenizer\n,\n'distilbert-base-uncased'\n)\n## Want BERT instead of distilBERT? Uncomment the following line:\n#model_class, tokenizer_class, pretrained_weights = (ppb.BertModel, ppb.BertTokenizer, 'bert-base-uncased')\n# Load pretrained model/tokenizer\ntokenizer\n=\ntokenizer_class\n.\nfrom_pretrained\n(\npretrained_weights\n)\nmodel\n=\nmodel_class\n.\nfrom_pretrained\n(\npretrained_weights\n)\nWe can now tokenize the dataset. Note that we’re going to do things a little differently here from the example above. The example above tokenized and processed only one sentence. Here, we’ll tokenize and process all sentences together as a batch (the notebook processes a smaller group of examples just for resource considerations, let’s say 2000 examples).\nTokenization\ntokenized\n=\ndf\n[\n0\n].\napply\n((\nlambda\nx\n:\ntokenizer\n.\nencode\n(\nx\n,\nadd_special_tokens\n=\nTrue\n)))\nThis turns every sentence into the list of ids.\nThe dataset is currently a list (or pandas Series/DataFrame) of lists. Before DistilBERT can process this as input, we’ll need to make all the vectors the same size by padding shorter sentences with the token id 0. You can refer to the notebook for the padding step, it’s basic python string and array manipulation.\nAfter the padding, we have a matrix/tensor that is ready to be passed to BERT:\nProcessing with DistilBERT\nWe now create an input tensor out of the padded token matrix, and send that to DistilBERT\ninput_ids\n=\ntorch\n.\ntensor\n(\nnp\n.\narray\n(\npadded\n))\nwith\ntorch\n.\nno_grad\n():\nlast_hidden_states\n=\nmodel\n(\ninput_ids\n)\nAfter running this step,\nlast_hidden_states\nholds the outputs of DistilBERT. It is a tuple with the shape (number of examples, max number of tokens in the sequence, number of hidden units in the DistilBERT model). In our case, this will be 2000 (since we only limited ourselves to 2000 examples), 66 (which is the number of tokens in the longest sequence from the 2000 examples), 768 (the number of hidden units in the DistilBERT model).\nUnpacking the BERT output tensor\nLet’s unpack this 3-d output tensor. We can first start by examining its dimensions:\nRecapping a sentence’s journey\nEach row is associated with a sentence from our dataset. To recap the processing path of the first sentence, we can think of it as looking like this:\nSlicing the important part\nFor sentence classification, we’re only only interested in BERT’s output for the [CLS] token, so we select that slice of the cube and discard everything else.\nThis is how we slice that 3d tensor to get the 2d tensor we’re interested in:\n# Slice the output for the first position for all the sequences, take all hidden unit outputs\nfeatures\n=\nlast_hidden_states\n[\n0\n][:,\n0\n,:].\nnumpy\n()\nAnd now\nfeatures\nis a 2d numpy array containing the sentence embeddings of all the sentences in our dataset.\nThe tensor we sliced from BERT's output\nDataset for Logistic Regression\nNow that we have the output of BERT, we have assembled the dataset we need to train our logistic regression model. The 768 columns are the features, and the labels we just get from our initial dataset.\nThe labeled dataset we use to train the Logistic Regression. The features are the output vectors of BERT for the [CLS] token (position #0) that we sliced in the previous figure. Each row corresponds to a sentence in our dataset, each column corresponds to the output of a hidden unit from the feed-forward neural network at the top transformer block of the Bert/DistilBERT model.\nAfter doing the traditional train/test split of machine learning, we can declare our Logistic Regression model and train it against the dataset.\nlabels\n=\ndf\n[\n1\n]\ntrain_features\n,\ntest_features\n,\ntrain_labels\n,\ntest_labels\n=\ntrain_test_split\n(\nfeatures\n,\nlabels\n)\nWhich splits the dataset into training/testing sets:\nNext, we train the Logistic Regression model on the training set.\nlr_clf\n=\nLogisticRegression\n()\nlr_clf\n.\nfit\n(\ntrain_features\n,\ntrain_labels\n)\nNow that the model is trained, we can score it against the test set:\nlr_clf\n.\nscore\n(\ntest_features\n,\ntest_labels\n)\nWhich shows the model achieves around 81% accuracy.\nScore Benchmarks\nFor reference, the highest accuracy score for this dataset is currently\n96.8\n. DistilBERT can be trained to improve its score on this task – a process called fine-tuning which updates BERT’s weights to make it achieve a better performance in the sentence classification (which we can call the\ndownstream task\n). The fine-tuned DistilBERT turns out to achieve an accuracy score of\n90.7\n. The full size BERT model achieves\n94.9\n.\nThe Notebook\nDive right into\nthe notebook\nor\nrun it on colab\n.\nAnd that’s it! That’s a good first contact with BERT. The next step would be to head over to the documentation and try your hand at\nfine-tuning\n. You can also go back and switch from distilBERT to BERT and see how that works.\nThanks to\nClément Delangue\n,\nVictor Sanh\n, and the Huggingface team for providing feedback to earlier versions of this tutorial.\nWritten on November 26, 2019",
  "metadata": {
    "title": "A Visual Guide to Using BERT for the First Time – Jay Alammar – Visualizing machine learning one concept at a time.",
    "description": "Translations: Chinese, Korean, Russian\n\n\n  \n  \n\n\nProgress has been rapidly accelerating in machine learning models that process language over the last couple of years. This progress has left the research lab and started powering some of the leading digital products. A great example of this is the recent announcement of how the BERT model is now a major force behind Google Search. Google believes this step (or progress in natural language understanding as applied in search) represents “the biggest leap forward in the past five years, and one of the biggest leaps forward in the history of Search”.\n\nThis post is a simple tutorial for how to use a variant of BERT to classify sentences. This is an example that is basic enough as a first intro, yet advanced enough to showcase some of the key concepts involved.\n\nAlongside this post, I’ve prepared a notebook. You can see it here the notebook or run it on colab.",
    "keywords": null,
    "author": "Jay Alammar",
    "og:description": "Translations: Chinese, Korean, Russian\n\n\n  \n  \n\n\nProgress has been rapidly accelerating in machine learning models that process language over the last couple of years. This progress has left the research lab and started powering some of the leading digital products. A great example of this is the recent announcement of how the BERT model is now a major force behind Google Search. Google believes this step (or progress in natural language understanding as applied in search) represents “the biggest leap forward in the past five years, and one of the biggest leaps forward in the history of Search”.\n\nThis post is a simple tutorial for how to use a variant of BERT to classify sentences. This is an example that is basic enough as a first intro, yet advanced enough to showcase some of the key concepts involved.\n\nAlongside this post, I’ve prepared a notebook. You can see it here the notebook or run it on colab.",
    "og:title": "A Visual Guide to Using BERT for the First Time"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2b93130a1e7d/code/body-000.txt",
      "chars": 84,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-001.txt",
      "chars": 259,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-002.txt",
      "chars": 141,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-003.txt",
      "chars": 10,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-004.txt",
      "chars": 455,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-005.txt",
      "chars": 82,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-006.txt",
      "chars": 108,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-007.txt",
      "chars": 143,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-008.txt",
      "chars": 110,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-009.txt",
      "chars": 71,
      "language": "text"
    },
    {
      "file": "attachments/2b93130a1e7d/code/body-010.txt",
      "chars": 41,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "http://www.junphy.com/wordpress/index.php/2020/10/20/a-visual-guide-using-bert/",
      "label": "Chinese",
      "type": "a"
    },
    {
      "href": "https://chloamme.github.io/2021/12/22/a-visual-guide-to-using-bert-for-the-first-time-korean.html",
      "label": "Korean",
      "type": "a"
    },
    {
      "href": "https://habr.com/ru/post/498144/",
      "label": "Russian",
      "type": "a"
    },
    {
      "href": "https://www.blog.google/products/search/search-language-understanding-bert/",
      "label": "recent announcement of how the BERT model is now a major force behind Google Search",
      "type": "a"
    },
    {
      "href": "https://github.com/jalammar/jalammar.github.io/blob/master/notebooks/bert/A_Visual_Notebook_to_Using_BERT_for_the_First_Time.ipynb",
      "label": "the notebook",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/github/jalammar/jalammar.github.io/blob/master/notebooks/bert/A_Visual_Notebook_to_Using_BERT_for_the_First_Time.ipynb",
      "label": "run it on colab",
      "type": "a"
    },
    {
      "href": "https://nlp.stanford.edu/sentiment/index.html",
      "label": "SST2",
      "type": "a"
    },
    {
      "href": "https://medium.com/huggingface/distilbert-8cf3380435b5",
      "label": "DistilBERT",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/",
      "label": "HuggingFace",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/illustrated-bert/",
      "label": "Illustrated BERT",
      "type": "a"
    },
    {
      "href": "https://github.com/huggingface/transformers",
      "label": "transformers",
      "type": "a"
    },
    {
      "href": "https://jalammar.github.io/illustrated-word2vec/",
      "label": "The Illustrated Word2vec",
      "type": "a"
    },
    {
      "href": "https://github.com/clairett/pytorch-sentiment-classification/",
      "label": "available",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/transformers/examples.html",
      "label": "fine-tuning",
      "type": "a"
    },
    {
      "href": "https://twitter.com/ClementDelangue",
      "label": "Clément Delangue",
      "type": "a"
    },
    {
      "href": "https://twitter.com/SanhEstPasMoi",
      "label": "Victor Sanh",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:29Z",
  "source_url": "https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11903,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2b93130a1e7d/pages/source.html",
  "local_html": "attachments/2b93130a1e7d/pages/article.html",
  "body_base_path": "attachments/2b93130a1e7d/pages/",
  "body_markdown_path": "attachments/2b93130a1e7d/pages/article.md"
}