{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "Tokenization and Sequencing in TensorFlow",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/tokenization-and-sequencing-in-tensorflow-3p0n",
  "publish_date": "2021-04-20",
  "corpus_stage": "",
  "notes": "",
  "id": "197d6bffee8e",
  "collected_at": "2026-10-02T14:01:41",
  "title": "Tokenization and Sequencing in TensorFlow - DEV Community",
  "body_markdown": "In this blog post, we shall seek to learn how to implement tokenization and sequencing, important text pre-processing steps, in Tensorflow.\nOutline\nIntroduction to Tokenizer\nUnderstanding Sequencing\nIntroduction to Tokenizer\nTokenization is the process of splitting the text into smaller units such as sentences, words or subwords. In this section, we shall see how we can pre-process the text corpus by tokenizing text into words in TensorFlow. We shall use the Keras API with TensorFlow backend; The code snippet below shows the necessary imports.\nimport\ntensorflow\nas\ntf\nfrom\ntensorflow\nimport\nkeras\nfrom\ntensorflow.keras.preprocessing.text\nimport\nTokenizer\nEnter fullscreen mode\nExit fullscreen mode\nAnd voila!🎉 we have all modules imported! Let’s initialize a list of sentences that we shall tokenize.\nsentences\n=\n[\n'\nLife is so beautiful\n'\n,\n'\nHope keeps us going\n'\n,\n'\nLet us celebrate life!\n'\n]\nEnter fullscreen mode\nExit fullscreen mode\nThe next step is to instantiate the Tokenizer and call the\nfit_on_texts\nmethod.\ntokenizer\n=\nTokenizer\n()\ntokenizer\n.\nfit_on_texts\n(\nsentences\n)\nEnter fullscreen mode\nExit fullscreen mode\nWell, when the text corpus is very large, we can specify an additional\nnum_words\nargument to get the most frequent words. For example, if we’d like to get the 100 most frequent words in the corpus, then\ntokenizer = Tokenizer(num_words=100)\ndoes just that!\nTo know how these tokens have been created and the indices assigned to words, we can use the\nword_index\nattribute.\nword_index\n=\ntokenizer\n.\nword_index\nprint\n(\nword_index\n)\n# Here’s the output:\n{\n‘\nlife\n’\n:\n1\n,\n‘\nus\n’\n:\n2\n,\n‘\nis\n’\n:\n3\n,\n‘\nso\n’\n:\n4\n,\n‘\nbeautiful\n’\n:\n5\n,\n‘\nhope\n’\n:\n6\n,\n‘\nkeeps\n’\n:\n7\n,\n‘\ngoing\n’\n:\n8\n,\n‘\nlet\n’\n:\n9\n,\n‘\ncelebrate\n’\n:\n10\n}\nEnter fullscreen mode\nExit fullscreen mode\nWell, so far so good! But what happens when the test data contains words that we’ve not accounted for in the vocabulary?🤔\ntest_data\n=\n[\n'\nOur life is to celebrate\n'\n,\n'\nHoping for the best!\n'\n,\n'\nLet peace prevail everywhere\n'\n]\nEnter fullscreen mode\nExit fullscreen mode\nWe have introduced sentences in\ntest_data\nwhich contain words that are not in our earlier vocabulary.\nHow do we account for such words which are not in vocabulary? We can define an argument\noov_token\nto account for such Out Of Vocabulary (OOV) tokens.\ntokenizer\n=\nTokenizer\n(\noov_token\n=\n”\n<\nOOV\n>\n”\n)\nThe\nword_index\nnow\nreturns\nthe\nfollowing\noutput\n:\n{\n‘\n<\nOOV\n>\n’\n:\n1\n,\n‘\nlife\n’\n:\n2\n,\n‘\nus\n’\n:\n3\n,\n‘\nis\n’\n:\n4\n,\n‘\nso\n’\n:\n5\n,\n‘\nbeautiful\n’\n:\n6\n,\n‘\nhope\n’\n:\n7\n,\n‘\nkeeps\n’\n:\n8\n,\n‘\ngoing\n’\n:\n9\n,\n‘\nlet\n’\n:\n10\n,\n‘\ncelebrate\n’\n:\n11\n}\nEnter fullscreen mode\nExit fullscreen mode\nUnderstanding Sequencing\nIn this section, we shall build on the tokenized text, using these generated tokens to convert the text into a sequence.\nWe can get a sequence by calling the\ntexts_to_sequences\nmethod.\nsequences\n=\ntokenizer\n.\ntexts_to_sequences\n(\nsentences\n)\n#Here’s the output:\n[[\n2\n,\n4\n,\n5\n,\n6\n],\n[\n7\n,\n8\n,\n3\n,\n9\n],\n[\n10\n,\n3\n,\n11\n,\n2\n]]\nEnter fullscreen mode\nExit fullscreen mode\nLet’s now take a step back. What happens when the sentences are of different lengths?Then, we will have to convert all of them to the same length.\nWe shall import\npad_sequences\nfunction to pad our sequences and look at the padded sequences.\nfrom\ntensorflow.keras.preprocessing.sequence\nimport\npad_sequences\npadded\n=\npad_sequences\n(\nsequences\n)\nprint\n(\n\"\n\\n\nPadded Sequences:\n\"\n)\nprint\n(\npadded\n)\n# Output\nPadded\nSequences\n:\n[[\n2\n4\n5\n6\n]\n[\n7\n8\n3\n9\n]\n[\n10\n3\n11\n2\n]]\nEnter fullscreen mode\nExit fullscreen mode\nBy default, the\nlength of the padded sequence = length of the longest sentence\n. However, we can limit the maximum length by explicitly setting the\nmaxlen\nargument.\npadded\n=\npad_sequences\n(\nsequences\n,\nmaxlen\n=\n5\n)\nprint\n(\n\"\n\\n\nPadded Sequences:\n\"\n)\nprint\n(\npadded\n)\n# Output\nPadded\nSequences\n:\n[[\n0\n2\n4\n5\n6\n]\n[\n0\n7\n8\n3\n9\n]\n[\n0\n10\n3\n11\n2\n]]\nEnter fullscreen mode\nExit fullscreen mode\nNow, let’s pad our test sequences after converting them to sequences.\ntest_seq\n=\ntokenizer\n.\ntexts_to_sequences\n(\ntest_data\n)\nprint\n(\n\"\n\\n\nTest Sequence =\n\"\n,\ntest_seq\n)\npadded\n=\npad_sequences\n(\ntest_seq\n,\nmaxlen\n=\n10\n)\nprint\n(\n\"\n\\n\nPadded Test Sequence:\n\"\n)\nprint\n(\npadded\n)\nAnd\nhere\n’\ns\nour\noutput\n.\n# Output\nTest\nSequence\n=\n[[\n1\n,\n2\n,\n4\n,\n1\n,\n11\n],\n[\n1\n,\n1\n,\n1\n,\n1\n],\n[\n10\n,\n1\n,\n1\n,\n1\n]]\nPadded\nTest\nSequence\n:\n[[\n0\n0\n0\n0\n0\n1\n2\n4\n1\n11\n]\n[\n0\n0\n0\n0\n0\n0\n1\n1\n1\n1\n]\n[\n0\n0\n0\n0\n0\n0\n10\n1\n1\n1\n]]\nEnter fullscreen mode\nExit fullscreen mode\nWe see that all the padded sequences are of length\nmaxlen\nand are padded with 0s at the beginning. What if we would like to add trailing zeros instead of at the beginning? We only need to specify\npadding='post'\npadded\n=\npad_sequences\n(\ntest_seq\n,\nmaxlen\n=\n10\n,\npadding\n=\n'\npost\n'\n)\nprint\n(\n\"\n\\n\nPadded Test Sequence:\n\"\n)\nprint\n(\npadded\n)\n# Output\nPadded\nTest\nSequence\n:\n[[\n1\n2\n4\n1\n11\n0\n0\n0\n0\n0\n]\n[\n1\n1\n1\n1\n0\n0\n0\n0\n0\n0\n]\n[\n10\n1\n1\n1\n0\n0\n0\n0\n0\n0\n]]\nEnter fullscreen mode\nExit fullscreen mode\nSo far, none of the sentences have length exceeding\nmaxlen\n, but in practice, we may have sentences that are much longer than\nmaxlen\n. In that case, we have to truncate the sentences and can set the argument\ntruncating='post'\nor\n'pre'\nto drop the first few or the last few words that exceed the specified\nmaxlen\n.\nHappy learning and coding!🎈✨🎉👩🏽‍💻\nReference\nNatural Language Processing in TensorFlow on Coursera\nCover Image: Photo by\nSusan Q Yin\non\nUnsplash",
  "metadata": {
    "title": "Tokenization and Sequencing in TensorFlow - DEV Community",
    "description": "In this blog post, we shall seek to learn how to implement tokenization and sequencing, important... Tagged with python, nlp, machinelearning.",
    "keywords": "python, nlp, machinelearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/tokenization-and-sequencing-in-tensorflow-3p0n",
    "og:title": "Tokenization and Sequencing in TensorFlow",
    "og:description": "In this blog post, we shall seek to learn how to implement tokenization and sequencing, important...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fycbjxgdmoxtk4snacdta.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Tokenization and Sequencing in TensorFlow",
    "twitter:description": "In this blog post, we shall seek to learn how to implement tokenization and sequencing, important...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fycbjxgdmoxtk4snacdta.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/197d6bffee8e/code/body-000.txt",
      "chars": 111,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-001.txt",
      "chars": 88,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-002.txt",
      "chars": 58,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-003.txt",
      "chars": 192,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-004.txt",
      "chars": 99,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-005.txt",
      "chars": 221,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-006.txt",
      "chars": 118,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-007.txt",
      "chars": 222,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-008.txt",
      "chars": 171,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-009.txt",
      "chars": 408,
      "language": "text"
    },
    {
      "file": "attachments/197d6bffee8e/code/body-010.txt",
      "chars": 244,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.coursera.org/learn/natural-language-processing-tensorflow",
      "label": "Natural Language Processing in TensorFlow on Coursera",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@syinq?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Susan Q Yin",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/s/photos/books?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:12Z",
  "source_url": "https://dev.to/balapriya/tokenization-and-sequencing-in-tensorflow-3p0n",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5387,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/197d6bffee8e/pages/source.html",
  "local_html": "attachments/197d6bffee8e/pages/article.html",
  "body_base_path": "attachments/197d6bffee8e/pages/",
  "body_markdown_path": "attachments/197d6bffee8e/pages/article.md"
}