{
  "source_file": "0930",
  "source_platform": "研究机构",
  "author": "Dive into Deep Learning",
  "title_listed": "Natural Language Processing: Pretraining（章节）",
  "keywords": "NLP,预训练,BERT",
  "domain": "文本处理",
  "content_type": "教程/书籍",
  "is_paid_flag": "否",
  "url": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/index.html",
  "publish_date": "持续更新",
  "corpus_stage": "文本处理",
  "notes": "NLP 预训练章节",
  "id": "66e52692cd99",
  "collected_at": "2026-10-02T16:07:43",
  "title": "15. Natural Language Processing: Pretraining — Dive into Deep Learning 1.0.3 documentation",
  "body_markdown": "Table Of Contents\n15.\nNatural Language Processing: Pretraining\n¶\nHumans need to communicate. Out of this basic need of the human\ncondition, a vast amount of written text has been generated on an\neveryday basis. Given rich text in social media, chat apps, emails,\nproduct reviews, news articles, research papers, and books, it becomes\nvital to enable computers to understand them to offer assistance or make\ndecisions based on human languages.\nNatural language processing\nstudies interactions between computers and\nhumans using natural languages. In practice, it is very common to use\nnatural language processing techniques to process and analyze text\n(human natural language) data, such as language models in\nSection 9.3\nand machine translation models in\nSection 10.5\n.\nTo understand text, we can begin by learning its representations.\nLeveraging the existing text sequences from large corpora,\nself-supervised learning\nhas been extensively used to pretrain text\nrepresentations, such as by predicting some hidden part of the text\nusing some other part of their surrounding text. In this way, models\nlearn through supervision from\nmassive\ntext data without\nexpensive\nlabeling efforts!\nAs we will see in this chapter, when treating each word or subword as an\nindividual token, the representation of each token can be pretrained\nusing word2vec, GloVe, or subword embedding models on large corpora.\nAfter pretraining, representation of each token can be a vector,\nhowever, it remains the same no matter what the context is. For\ninstance, the vector representation of “bank” is the same in both “go to\nthe bank to deposit some money” and “go to the bank to sit down”. Thus,\nmany more recent pretraining models adapt representation of the same\ntoken to different contexts. Among them is BERT, a much deeper\nself-supervised model based on the Transformer encoder. In this chapter,\nwe will focus on how to pretrain such representations for text, as\nhighlighted in\nFig. 15.1\n.\nFig. 15.1\nPretrained text representations can be fed to various deep learning\narchitectures for different downstream natural language processing\napplications. This chapter focuses on the upstream text\nrepresentation pretraining.\n¶\nFor sight of the big picture,\nFig. 15.1\nshows\nthat the pretrained text representations can be fed to a variety of deep\nlearning architectures for different downstream natural language\nprocessing applications. We will cover them in\nSection 16\n.\n15.1. Word Embedding (word2vec)\n15.1.1. One-Hot Vectors Are a Bad Choice\n15.1.2. Self-Supervised word2vec\n15.1.3. The Skip-Gram Model\n15.1.4. The Continuous Bag of Words (CBOW) Model\n15.1.5. Summary\n15.1.6. Exercises\n15.2. Approximate Training\n15.2.1. Negative Sampling\n15.2.2. Hierarchical Softmax\n15.2.3. Summary\n15.2.4. Exercises\n15.3. The Dataset for Pretraining Word Embeddings\n15.3.1. Reading the Dataset\n15.3.2. Subsampling\n15.3.3. Extracting Center Words and Context Words\n15.3.4. Negative Sampling\n15.3.5. Loading Training Examples in Minibatches\n15.3.6. Putting It All Together\n15.3.7. Summary\n15.3.8. Exercises\n15.4. Pretraining word2vec\n15.4.1. The Skip-Gram Model\n15.4.2. Training\n15.4.3. Applying Word Embeddings\n15.4.4. Summary\n15.4.5. Exercises\n15.5. Word Embedding with Global Vectors (GloVe)\n15.5.1. Skip-Gram with Global Corpus Statistics\n15.5.2. The GloVe Model\n15.5.3. Interpreting GloVe from the Ratio of Co-occurrence Probabilities\n15.5.4. Summary\n15.5.5. Exercises\n15.6. Subword Embedding\n15.6.1. The fastText Model\n15.6.2. Byte Pair Encoding\n15.6.3. Summary\n15.6.4. Exercises\n15.7. Word Similarity and Analogy\n15.7.1. Loading Pretrained Word Vectors\n15.7.2. Applying Pretrained Word Vectors\n15.7.3. Summary\n15.7.4. Exercises\n15.8. Bidirectional Encoder Representations from Transformers (BERT)\n15.8.1. From Context-Independent to Context-Sensitive\n15.8.2. From Task-Specific to Task-Agnostic\n15.8.3. BERT: Combining the Best of Both Worlds\n15.8.4. Input Representation\n15.8.5. Pretraining Tasks\n15.8.6. Putting It All Together\n15.8.7. Summary\n15.8.8. Exercises\n15.9. The Dataset for Pretraining BERT\n15.9.1. Defining Helper Functions for Pretraining Tasks\n15.9.2. Transforming Text into the Pretraining Dataset\n15.9.3. Summary\n15.9.4. Exercises\n15.10. Pretraining BERT\n15.10.1. Pretraining BERT\n15.10.2. Representing Text with BERT\n15.10.3. Summary\n15.10.4. Exercises\nPrevious\n14.14. Dog Breed Identification (ImageNet Dogs) on Kaggle\nNext\n15.1. Word Embedding (word2vec)",
  "metadata": {
    "title": "15. Natural Language Processing: Pretraining — Dive into Deep Learning 1.0.3 documentation",
    "description": null,
    "keywords": null,
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://www.d2l.ai/_static/logo-with-text.png",
      "bytes": 87175,
      "reused": false,
      "local_path": "attachments/66e52692cd99/images/body-509f124ba61493e64845.png",
      "src": "https://www.d2l.ai/_static/logo-with-text.png",
      "alt": "Dive into Deep Learning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://www.d2l.ai/_images/nlp-map-pretrain.svg",
      "bytes": 165586,
      "reused": false,
      "local_path": "attachments/66e52692cd99/images/body-9dc5cc7cbee70f603c0f.svg",
      "src": "https://www.d2l.ai/_images/nlp-map-pretrain.svg",
      "alt": "../_images/nlp-map-pretrain.svg",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.d2l.ai/index.html",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-neural-networks/language-model.html",
      "label": "Section 9.3",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_recurrent-modern/machine-translation-and-dataset.html",
      "label": "Section 10.5",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-applications/index.html",
      "label": "Section 16",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/word2vec.html",
      "label": "15.1. Word Embedding (word2vec)",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/approx-training.html",
      "label": "15.2. Approximate Training",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/word-embedding-dataset.html",
      "label": "15.3. The Dataset for Pretraining Word Embeddings",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/word2vec-pretraining.html",
      "label": "15.4. Pretraining word2vec",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/glove.html",
      "label": "15.5. Word Embedding with Global Vectors (GloVe)",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/subword-embedding.html",
      "label": "15.6. Subword Embedding",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/similarity-analogy.html",
      "label": "15.7. Word Similarity and Analogy",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/bert.html",
      "label": "15.8. Bidirectional Encoder Representations from Transformers (BERT)",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/bert-dataset.html",
      "label": "15.9. The Dataset for Pretraining BERT",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html",
      "label": "15.10. Pretraining BERT",
      "type": "a"
    },
    {
      "href": "https://www.d2l.ai/chapter_computer-vision/kaggle-dog.html",
      "label": "Previous 14.14. Dog Breed Identification (ImageNet Dogs) on Kaggle",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:48Z",
  "source_url": "https://www.d2l.ai/chapter_natural-language-processing-pretraining/index.html",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4440,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/66e52692cd99/pages/source.html",
  "local_html": "attachments/66e52692cd99/pages/article.html",
  "body_base_path": "attachments/66e52692cd99/pages/",
  "body_markdown_path": "attachments/66e52692cd99/pages/article.md"
}