{ "source_file": "0930", "source_platform": "技术平台", "author": "Jason Brownlee / Machine Learning Mastery", "title_listed": "How to Prepare Text Data for Machine Learning with scikit-learn", "keywords": "文本数据,scikit-learn,预处理", "domain": "文本处理", "content_type": "教程", "is_paid_flag": "否", "url": "https://machinelearningmastery.com/prepare-text-data-machine-learning-scikit-learn/", "publish_date": "2020-05", "corpus_stage": "数据处理", "notes": "用scikit-learn准备文本数据", "id": "2e61e566f7ab", "collected_at": "2026-10-02T17:55:04", "title": "How to Encode Text Data for Machine Learning with scikit-learn - MachineLearningMastery.com", "body_markdown": "How to Encode Text Data for Machine Learning with scikit-learn\nBy\nJason Brownlee\non\nJune 28, 2020\nin\nDeep Learning for Natural Language Processing\n119\nShare\nPost\nShare\nText data requires special preparation before you can start using it for predictive modeling.\nThe text must be parsed to remove words, called tokenization. Then the words need to be encoded as integers or floating point values for use as input to a machine learning algorithm, called feature extraction (or vectorization).\nThe scikit-learn library offers easy-to-use tools to perform both tokenization and feature extraction of your text data.\nIn this tutorial, you will discover exactly how you can prepare your text data for predictive modeling in Python with scikit-learn.\nAfter completing this tutorial, you will know:\nHow to convert text to word count vectors with CountVectorizer.\nHow to convert text to word frequency vectors with TfidfVectorizer.\nHow to convert text to unique integers with HashingVectorizer.\nKick-start your project\nwith my new book\nDeep Learning for Natural Language Processing\n, including\nstep-by-step tutorials\nand the\nPython source code\nfiles for all examples.\nLet’s get started.\nHow to Prepare Text Data for Machine Learning with scikit-learn\nPhoto by\nMartin Kelly\n, some rights reserved.\nBag-of-Words Model\nWe cannot work with text directly when using machine learning algorithms.\nInstead, we need to convert the text to numbers.\nWe may want to perform classification of documents, so each document is an “\ninput\n” and a class label is the “\noutput\n” for our predictive algorithm. Algorithms take vectors of numbers as input, therefore we need to convert documents to fixed-length vectors of numbers.\nA simple and effective model for thinking about text documents in machine learning is called the\nBag-of-Words Model\n, or\nBoW\n.\nThe model is simple in that it throws away all of the order information in the words and focuses on the occurrence of words in a document.\nThis can be done by assigning each word a unique number. Then any document we see can be encoded as a fixed-length vector with the length of the vocabulary of known words. The value in each position in the vector could be filled with a count or frequency of each word in the encoded document.\nThis is the bag of words model, where we are only concerned with encoding schemes that represent what words are present or the degree to which they are present in encoded documents without any information about order.\nFor more on the bag of words model, see the tutorial:\nA Gentle Introduction to the Bag-of-Words Model\nThere are many ways to extend this simple method, both by better clarifying what a “\nword\n” is and in defining what to encode about each word in the vector.\nThe scikit-learn library provides 3 different schemes that we can use, and we will briefly look at each.\nNeed help with Deep Learning for Text Data?\nTake my free 7-day email crash course now (with code).\nClick to sign-up and also get a free PDF Ebook version of the course.\nStart Your FREE Crash-Course Now\nWord Counts with CountVectorizer\nThe\nCountVectorizer\nprovides a simple way to both tokenize a collection of text documents and build a vocabulary of known words, but also to encode new documents using that vocabulary.\nYou can use it as follows:\nCreate an instance of the\nCountVectorizer\nclass.\nCall the\nfit()\nfunction in order to learn a vocabulary from one or more documents.\nCall the\ntransform()\nfunction on one or more documents as needed to encode each as a vector.\nAn encoded vector is returned with a length of the entire vocabulary and an integer count for the number of times each word appeared in the document.\nBecause these vectors will contain a lot of zeros, we call them sparse. Python provides an efficient way of handling sparse vectors in the\nscipy.sparse\npackage.\nThe vectors returned from a call to transform() will be sparse vectors, and you can transform them back to numpy arrays to look and better understand what is going on by calling the toarray() function.\nBelow is an example of using the CountVectorizer to tokenize, build a vocabulary, and then encode a document.\nfrom sklearn.feature_extraction.text import CountVectorizer\n# list of text documents\ntext = [\"The quick brown fox jumped over the lazy dog.\"]\n# create the transform\nvectorizer = CountVectorizer()\n# tokenize and build vocab\nvectorizer.fit(text)\n# summarize\nprint(vectorizer.vocabulary_)\n# encode document\nvector = vectorizer.transform(text)\n# summarize encoded vector\nprint(vector.shape)\nprint(type(vector))\nprint(vector.toarray())\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\nfrom\nsklearn\n.\nfeature_extraction\n.\ntext\nimport\nCountVectorizer\n# list of text documents\ntext\n=\n[\n\"The quick brown fox jumped over the lazy dog.\"\n]\n# create the transform\nvectorizer\n=\nCountVectorizer\n(\n)\n# tokenize and build vocab\nvectorizer\n.\nfit\n(\ntext\n)\n# summarize\nprint\n(\nvectorizer\n.\nvocabulary_\n)\n# encode document\nvector\n=\nvectorizer\n.\ntransform\n(\ntext\n)\n# summarize encoded vector\nprint\n(\nvector\n.\nshape\n)\nprint\n(\ntype\n(\nvector\n)\n)\nprint\n(\nvector\n.\ntoarray\n(\n)\n)\nAbove, you can see that we access the vocabulary to see what exactly was tokenized by calling:\nprint(vectorizer.vocabulary_)\n1\nprint\n(\nvectorizer\n.\nvocabulary_\n)\nWe can see that all words were made lowercase by default and that the punctuation was ignored. These and other aspects of tokenizing can be configured and I encourage you to review all of the options in the\nAPI documentation\n.\nRunning the example first prints the vocabulary, then the shape of the encoded document. We can see that there are 8 words in the vocab, and therefore encoded vectors have a length of 8.\nWe can then see that the encoded vector is a sparse matrix. Finally, we can see an array version of the encoded vector showing a count of 1 occurrence for each word except the (index and id 7) that has an occurrence of 2.\n{'dog': 1, 'fox': 2, 'over': 5, 'brown': 0, 'quick': 6, 'the': 7, 'lazy': 4, 'jumped': 3}\n(1, 8)\n\n[[1 1 1 1 1 1 1 2]]\n1\n2\n3\n4\n{'dog': 1, 'fox': 2, 'over': 5, 'brown': 0, 'quick': 6, 'the': 7, 'lazy': 4, 'jumped': 3}\n(1, 8)\n\n[[1 1 1 1 1 1 1 2]]\nImportantly, the same vectorizer can be used on documents that contain words not included in the vocabulary. These words are ignored and no count is given in the resulting vector.\nFor example, below is an example of using the vectorizer above to encode a document with one word in the vocab and one word that is not.\n# encode another document\ntext2 = [\"the puppy\"]\nvector = vectorizer.transform(text2)\nprint(vector.toarray())\n1\n2\n3\n4\n# encode another document\ntext2\n=\n[\n\"the puppy\"\n]\nvector\n=\nvectorizer\n.\ntransform\n(\ntext2\n)\nprint\n(\nvector\n.\ntoarray\n(\n)\n)\nRunning this example prints the array version of the encoded sparse vector showing one occurrence of the one word in the vocab and the other word not in the vocab completely ignored.\n[[0 0 0 0 0 0 0 1]]\n1\n[[0 0 0 0 0 0 0 1]]\nThe encoded vectors can then be used directly with a machine learning algorithm.\nWord Frequencies with TfidfVectorizer\nWord counts are a good starting point, but are very basic.\nOne issue with simple counts is that some words like “\nthe\n” will appear many times and their large counts will not be very meaningful in the encoded vectors.\nAn alternative is to calculate word frequencies, and by far the most popular method is called\nTF-IDF\n. This is an acronym than stands for “\nTerm Frequency – Inverse Document\n” Frequency which are the components of the resulting scores assigned to each word.\nTerm Frequency\n: This summarizes how often a given word appears within a document.\nInverse Document Frequency\n: This downscales words that appear a lot across documents.\nWithout going into the math, TF-IDF are word frequency scores that try to highlight words that are more interesting, e.g. frequent in a document but not across documents.\nThe\nTfidfVectorizer\nwill tokenize documents, learn the vocabulary and inverse document frequency weightings, and allow you to encode new documents. Alternately, if you already have a learned CountVectorizer, you can use it with a\nTfidfTransformer\nto just calculate the inverse document frequencies and start encoding documents.\nThe same create, fit, and transform process is used as with the CountVectorizer.\nBelow is an example of using the TfidfVectorizer to learn vocabulary and inverse document frequencies across 3 small documents and then encode one of those documents.\nfrom sklearn.feature_extraction.text import TfidfVectorizer\n# list of text documents\ntext = [\"The quick brown fox jumped over the lazy dog.\",\n\t\t\"The dog.\",\n\t\t\"The fox\"]\n# create the transform\nvectorizer = TfidfVectorizer()\n# tokenize and build vocab\nvectorizer.fit(text)\n# summarize\nprint(vectorizer.vocabulary_)\nprint(vectorizer.idf_)\n# encode document\nvector = vectorizer.transform([text[0]])\n# summarize encoded vector\nprint(vector.shape)\nprint(vector.toarray())\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\nfrom\nsklearn\n.\nfeature_extraction\n.\ntext\nimport\nTfidfVectorizer\n# list of text documents\ntext\n=\n[\n\"The quick brown fox jumped over the lazy dog.\"\n,\n\"The dog.\"\n,\n\"The fox\"\n]\n# create the transform\nvectorizer\n=\nTfidfVectorizer\n(\n)\n# tokenize and build vocab\nvectorizer\n.\nfit\n(\ntext\n)\n# summarize\nprint\n(\nvectorizer\n.\nvocabulary_\n)\nprint\n(\nvectorizer\n.\nidf_\n)\n# encode document\nvector\n=\nvectorizer\n.\ntransform\n(\n[\ntext\n[\n0\n]\n]\n)\n# summarize encoded vector\nprint\n(\nvector\n.\nshape\n)\nprint\n(\nvector\n.\ntoarray\n(\n)\n)\nA vocabulary of 8 words is learned from the documents and each word is assigned a unique integer index in the output vector.\nThe inverse document frequencies are calculated for each word in the vocabulary, assigning the lowest score of 1.0 to the most frequently observed word: “\nthe\n” at index 7.\nFinally, the first document is encoded as an 8-element sparse array and we can review the final scorings of each word with different values for “\nthe\n“, “\nfox\n“, and “\ndog\n” from the other words in the vocabulary.\n{'fox': 2, 'lazy': 4, 'dog': 1, 'quick': 6, 'the': 7, 'over': 5, 'brown': 0, 'jumped': 3}\n[ 1.69314718 1.28768207 1.28768207 1.69314718 1.69314718 1.69314718\n1.69314718 1. ]\n(1, 8)\n[[ 0.36388646 0.27674503 0.27674503 0.36388646 0.36388646 0.36388646\n0.36388646 0.42983441]]\n1\n2\n3\n4\n5\n6\n{'fox': 2, 'lazy': 4, 'dog': 1, 'quick': 6, 'the': 7, 'over': 5, 'brown': 0, 'jumped': 3}\n[ 1.69314718 1.28768207 1.28768207 1.69314718 1.69314718 1.69314718\n1.69314718 1. ]\n(1, 8)\n[[ 0.36388646 0.27674503 0.27674503 0.36388646 0.36388646 0.36388646\n0.36388646 0.42983441]]\nThe scores are normalized to values between 0 and 1 and the encoded document vectors can then be used directly with most machine learning algorithms.\nHashing with HashingVectorizer\nCounts and frequencies can be very useful, but one limitation of these methods is that the vocabulary can become very large.\nThis, in turn, will require large vectors for encoding documents and impose large requirements on memory and slow down algorithms.\nA clever work around is to use a one way hash of words to convert them to integers. The clever part is that no vocabulary is required and you can choose an arbitrary-long fixed length vector. A downside is that the hash is a one-way function so there is no way to convert the encoding back to a word (which may not matter for many supervised learning tasks).\nThe\nHashingVectorizer\nclass implements this approach that can be used to consistently hash words, then tokenize and encode documents as needed.\nThe example below demonstrates the HashingVectorizer for encoding a single document.\nAn arbitrary fixed-length vector size of 20 was chosen. This corresponds to the range of the hash function, where small values (like 20) may result in hash collisions. Remembering back to compsci classes, I believe there are heuristics that you can use to pick the hash length and probability of collision based on estimated vocabulary size.\nNote that this vectorizer does not require a call to fit on the training data documents. Instead, after instantiation, it can be used directly to start encoding documents.\nfrom sklearn.feature_extraction.text import HashingVectorizer\n# list of text documents\ntext = [\"The quick brown fox jumped over the lazy dog.\"]\n# create the transform\nvectorizer = HashingVectorizer(n_features=20)\n# encode document\nvector = vectorizer.transform(text)\n# summarize encoded vector\nprint(vector.shape)\nprint(vector.toarray())\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\nfrom\nsklearn\n.\nfeature_extraction\n.\ntext\nimport\nHashingVectorizer\n# list of text documents\ntext\n=\n[\n\"The quick brown fox jumped over the lazy dog.\"\n]\n# create the transform\nvectorizer\n=\nHashingVectorizer\n(\nn_features\n=\n20\n)\n# encode document\nvector\n=\nvectorizer\n.\ntransform\n(\ntext\n)\n# summarize encoded vector\nprint\n(\nvector\n.\nshape\n)\nprint\n(\nvector\n.\ntoarray\n(\n)\n)\nRunning the example encodes the sample document as a 20-element sparse array.\nThe values of the encoded document correspond to normalized word counts by default in the range of -1 to 1, but could be made simple integer counts by changing the default configuration.\n(1, 20)\n[[ 0. 0. 0. 0. 0. 0.33333333\n 0. -0.33333333 0.33333333 0. 0. 0.33333333\n 0. 0. 0. -0.33333333 0. 0.\n -0.66666667 0. ]]\n1\n2\n3\n4\n5\n(1, 20)\n[[ 0.          0.          0.          0.          0.          0.33333333\n0.         -0.33333333  0.33333333  0.          0.          0.33333333\n0.          0.          0.         -0.33333333  0.          0.\n-0.66666667  0.        ]]\nFurther Reading\nThis section provides more resources on the topic if you are looking go deeper.\nNatural Language Processing\nBag-of-words model on Wikipedia\nTokenization on Wikipedia\nTF-IDF on Wikipedia\nsciki-learn\nSection 4.2. Feature extraction\n, scikit-learn User Guide\nsckit-learn Feature Extraction API\nWorking With Text Data\n, scikit-learn Tutorial\nClass APIs\nCountVectorizer scikit-learn API\nTfidfVectorizer scikit-learn API\nTfidfTransformer scikit-learn API\nHashingVectorizer scikit-learn API\nSummary\nIn this tutorial, you discovered how to prepare text documents for machine learning with scikit-learn.\nWe have only scratched the surface in these examples and I want to highlight that there are many configuration details for these classes to influence the tokenizing of documents that are worth exploring.\nDo you have any questions?\nAsk your questions in the comments below and I will do my best to answer.\nDevelop Deep Learning models for Text Data Today!\nDevelop Your Own Text models in Minutes\n...with just a few lines of python code\nDiscover how in my new Ebook:\nDeep Learning for Natural Language Processing\nIt provides\nself-study tutorials\non topics like:\nBag-of-Words, Word Embedding, Language Models, Caption Generation, Text Translation\nand much more...\nFinally Bring Deep Learning to your Natural Language Processing Projects\nSkip the Academics. Just Results.\nSee What's Inside\nShare\nPost\nShare\nMore On This Topic\nWhy One-Hot Encode Data in Machine Learning?\nHow to One Hot Encode Sequence Data in Python\n3 Smart Ways to Encode Categorical Features for…\n3 Ways to Encode Categorical Variables for Deep Learning\nHow to Prepare Text Data for Deep Learning with Keras\nFrom Text to Tables: Feature Engineering with LLMs…\nAbout Jason Brownlee\nJason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with modern machine learning methods via hands-on tutorials.\nView all posts by Jason Brownlee\n→", "metadata": { "title": "How to Encode Text Data for Machine Learning with scikit-learn - MachineLearningMastery.com", "description": null, "keywords": null, "author": "Jason Brownlee", "og:locale": "en_US", "og:type": "article", "og:title": "How to Encode Text Data for Machine Learning with scikit-learn - MachineLearningMastery.com", "og:description": "Text data requires special preparation before you can start using it for predictive modeling. The text must be parsed to remove words, called tokenization. Then the words need to be encoded as integers or floating point values for use as input to a machine learning algorithm, called feature extraction (or vectorization). The scikit-learn library offers easy-to-use tools to perform both…", "og:url": "https://machinelearningmastery.com/prepare-text-data-machine-learning-scikit-learn/", "og:site_name": "MachineLearningMastery.com", "og:image": "https://machinelearningmastery.com/wp-content/uploads/2017/09/How-to-Prepare-Text-Data-for-Machine-Learning-with-scikit-learn.jpg", "og:image:width": "640", "og:image:height": "427", "og:image:type": "image/jpeg", "twitter:label1": "Written by", "twitter:data1": "Jason Brownlee", "twitter:label2": "Est. reading time", "twitter:data2": "9 minutes", "twitter:card": "summary_large_image", "twitter:description": "Text data requires special preparation before you can start using it for predictive modeling. The text must be parsed to remove words, called tokenization. Then the words need to be encoded as integers or floating point values for use as input to a machine learning algorithm, called feature extraction (or vectorization). The scikit-learn library offers easy-to-use tools to perform both…", "twitter:title": "How to Encode Text Data for Machine Learning with scikit-learn - MachineLearningMastery.com", "article:publisher": "https://www.facebook.com/MachineLearningMastery/", "article:author": "https://www.facebook.com/MachineLearningMastery/", "article:published_time": "2017-09-28T19:00:18+00:00", "article:modified_time": "2020-06-27T19:40:54+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2017/09/How-to-Prepare-Text-Data-for-Machine-Learning-with-scikit-learn.jpg", "bytes": 56900, "reused": true, "local_path": "attachments/2e61e566f7ab/images/000.jpg", "src": "https://machinelearningmastery.com/wp-content/uploads/2017/09/How-to-Prepare-Text-Data-for-Machine-Learning-with-scikit-learn.jpg", "alt": "How to Prepare Text Data for Machine Learning with scikit-learn", "downloaded": true }, { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4NLP-220.png", "bytes": 4110, "reused": false, "local_path": "attachments/2e61e566f7ab/images/body-e8fcb21cfc1be3513af1.png", "src": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4NLP-220.png", "alt": "Deep Learning for Natural Language Processing", "downloaded": true }, { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2017/07/Why-One-Hot-Encode-Data-in-Machine-Learning.jpg", "bytes": 67650, "reused": true, "local_path": "attachments/2e61e566f7ab/images/003.jpg", "src": "https://machinelearningmastery.com/wp-content/uploads/2017/07/Why-One-Hot-Encode-Data-in-Machine-Learning.jpg", "alt": "Why One-Hot Encode Data in Machine Learning?", "downloaded": true }, { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2017/07/How-to-One-Hot-Encode-Sequence-Classification-Data-in-Python.jpg", "bytes": 89025, "reused": true, "local_path": "attachments/2e61e566f7ab/images/005.jpg", "src": "https://machinelearningmastery.com/wp-content/uploads/2017/07/How-to-One-Hot-Encode-Sequence-Classification-Data-in-Python.jpg", "alt": "How to One Hot Encode Sequence Classification Data in Python", "downloaded": true }, { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2025/12/mlm-3-smart-ways-encode-categorical-features-machine-learning-feature-200x200.jpeg", "bytes": 12082, "reused": true, "local_path": "attachments/2e61e566f7ab/images/007.jpeg", "src": "https://machinelearningmastery.com/wp-content/uploads/2025/12/mlm-3-smart-ways-encode-categorical-features-machine-learning-feature-200x200.jpeg", "alt": "mlm-3-smart-ways-encode-categorical-features-machine-learning-feature", "downloaded": true }, { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2019/11/How-to-Encode-Categorical-Data-for-Deep-Learning-in-Keras.jpg", "bytes": 46662, "reused": true, "local_path": "attachments/2e61e566f7ab/images/009.jpg", "src": "https://machinelearningmastery.com/wp-content/uploads/2019/11/How-to-Encode-Categorical-Data-for-Deep-Learning-in-Keras.jpg", "alt": "How to Encode Categorical Data for Deep Learning in Keras", "downloaded": true }, { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2017/10/How-to-Prepare-Text-Data-for-Deep-Learning-with-Keras.jpg", "bytes": 80486, "reused": true, "local_path": "attachments/2e61e566f7ab/images/011.jpg", "src": "https://machinelearningmastery.com/wp-content/uploads/2017/10/How-to-Prepare-Text-Data-for-Deep-Learning-with-Keras.jpg", "alt": "How to Prepare Text Data for Deep Learning with Keras", "downloaded": true }, { "status": "ok", "url": "https://machinelearningmastery.com/wp-content/uploads/2026/02/mlm-turning-raw-text-into-ml-features-with-llms-200x200.png", "bytes": 63430, "reused": true, "local_path": "attachments/2e61e566f7ab/images/013.png", "src": "https://machinelearningmastery.com/wp-content/uploads/2026/02/mlm-turning-raw-text-into-ml-features-with-llms-200x200.png", "alt": "mlm-turning-raw-text-into-ml-features-with-llms", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://machinelearningmastery.com/author/jasonb/", "label": "Jason Brownlee", "type": "a" }, { "href": "https://machinelearningmastery.com/category/natural-language-processing/", "label": "Deep Learning for Natural Language Processing", "type": "a" }, { "href": "https://machinelearningmastery.com/prepare-text-data-machine-learning-scikit-learn/", "label": "119", "type": "a" }, { "href": "https://machinelearningmastery.com/deep-learning-for-nlp/", "label": "Deep Learning for Natural Language Processing", "type": "a" }, { "href": "https://www.flickr.com/photos/martkelly/34474622542/", "label": "Martin Kelly", "type": "a" }, { "href": "https://machinelearningmastery.com/gentle-introduction-bag-words-model/", "label": "A Gentle Introduction to the Bag-of-Words Model", "type": "a" }, { "href": "http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html", "label": "CountVectorizer", "type": "a" }, { "href": "https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html", "label": "scipy.sparse", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf", "label": "TF-IDF", "type": "a" }, { "href": "http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html", "label": "TfidfVectorizer", "type": "a" }, { "href": "http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html", "label": "TfidfTransformer", "type": "a" }, { "href": "http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html", "label": "HashingVectorizer", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Bag-of-words_model", "label": "Bag-of-words model on Wikipedia", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Lexical_analysis", "label": "Tokenization on Wikipedia", "type": "a" }, { "href": "http://scikit-learn.org/stable/modules/feature_extraction.html", "label": "Section 4.2. Feature extraction", "type": "a" }, { "href": "http://scikit-learn.org/stable/modules/classes.html", "label": "sckit-learn Feature Extraction API", "type": "a" }, { "href": "http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html", "label": "Working With Text Data", "type": "a" }, { "href": "https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/", "label": "Why One-Hot Encode Data in Machine Learning?", "type": "a" }, { "href": "https://machinelearningmastery.com/how-to-one-hot-encode-sequence-data-in-python/", "label": "How to One Hot Encode Sequence Data in Python", "type": "a" }, { "href": "https://machinelearningmastery.com/3-smart-ways-to-encode-categorical-features-for-machine-learning/", "label": "3 Smart Ways to Encode Categorical Features for…", "type": "a" }, { "href": "https://machinelearningmastery.com/how-to-prepare-categorical-data-for-deep-learning-in-python/", "label": "3 Ways to Encode Categorical Variables for Deep Learning", "type": "a" }, { "href": "https://machinelearningmastery.com/prepare-text-data-deep-learning-keras/", "label": "How to Prepare Text Data for Deep Learning with Keras", "type": "a" }, { "href": "https://machinelearningmastery.com/from-text-to-tables-feature-engineering-with-llms-for-tabular-data/", "label": "From Text to Tables: Feature Engineering with LLMs…", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:54:27Z", "source_url": "https://machinelearningmastery.com/prepare-text-data-machine-learning-scikit-learn/", "papers": [], "actual_video_files": [], "body_chars": 15553, "expected_body_images": 8, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/2e61e566f7ab/pages/source.html", "local_html": "attachments/2e61e566f7ab/pages/article.html", "body_base_path": "attachments/2e61e566f7ab/pages/", "body_markdown_path": "attachments/2e61e566f7ab/pages/article.md" }