{
  "source_file": "0930",
  "source_platform": "技术平台",
  "author": "Jason Brownlee / Machine Learning Mastery",
  "title_listed": "How to Clean Text for Machine Learning",
  "keywords": "文本清洗,NLP,预处理",
  "domain": "文本处理",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://machinelearningmastery.com/clean-text-machine-learning-python/",
  "publish_date": "2019-10",
  "corpus_stage": "数据处理",
  "notes": "机器学习文本清洗",
  "id": "59767026df5a",
  "collected_at": "2026-10-02T17:54:58",
  "title": "How to Clean Text for Machine Learning with Python - MachineLearningMastery.com",
  "body_markdown": "How to Clean Text for Machine Learning with Python\nBy\nJason Brownlee\non\nAugust 7, 2019\nin\nDeep Learning for Natural Language Processing\n146\nShare\nPost\nShare\nYou cannot go straight from raw text to fitting a machine learning or deep learning model.\nYou must clean your text first, which means splitting it into words and handling punctuation and case.\nIn fact, there is a whole suite of text preparation methods that you may need to use, and the choice of methods really depends on your natural language processing task.\nIn this tutorial, you will discover how you can clean and prepare your text ready for modeling with machine learning.\nAfter completing this tutorial, you will know:\nHow to get started by developing your own very simple text cleaning tools.\nHow to take a step up and use the more sophisticated methods in the NLTK library.\nHow to prepare text when using modern text representation methods like word embeddings.\nKick-start your project\nwith my new book\nDeep Learning for Natural Language Processing\n, including\nstep-by-step tutorials\nand the\nPython source code\nfiles for all examples.\nLet’s get started.\nUpdate Nov/2017\n: Fixed a code typo in the ‘split into words’ section, thanks David Comfort.\nHow to Develop Multilayer Perceptron Models for Time Series Forecasting\nPhoto by Bureau of Land Management, some rights reserved.\nTutorial Overview\nThis tutorial is divided into 6 parts; they are:\nMetamorphosis by Franz Kafka\nText Cleaning is Task Specific\nManual Tokenization\nTokenization and Cleaning with NLTK\nAdditional Text Cleaning Considerations\nTips for Cleaning Text for Word Embedding\nNeed help with Deep Learning for Text Data?\nTake my free 7-day email crash course now (with code).\nClick to sign-up and also get a free PDF Ebook version of the course.\nStart Your FREE Crash-Course Now\nMetamorphosis by Franz Kafka\nLet’s start off by selecting a dataset.\nIn this tutorial, we will use the text from the book\nMetamorphosis\nby\nFranz Kafka\n. No specific reason, other than it’s short, I like it, and you may like it too. I expect it’s one of those classics that most students have to read in school.\nThe full text for Metamorphosis is available for free from Project Gutenberg.\nMetamorphosis by Franz Kafka on Project Gutenberg\nYou can download the ASCII text version of the text here:\nMetamorphosis by Franz Kafka Plain Text UTF-8\n(may need to load the page twice).\nDownload the file and place it in your current working directory with the file name “\nmetamorphosis.txt\n“.\nThe file contains header and footer information that we are not interested in, specifically copyright and license information. Open the file and delete the header and footer information and save the file as “\nmetamorphosis_clean.txt\n“.\nThe start of the clean file should look like:\nOne morning, when Gregor Samsa woke from troubled dreams, he found himself transformed in his bed into a horrible vermin.\nThe file should end with:\nAnd, as if in confirmation of their new dreams and good intentions, as soon as they reached their destination Grete was the first to get up and stretch out her young body.\nPoor Gregor…\nText Cleaning Is Task Specific\nAfter actually getting a hold of your text data, the first step in cleaning up text data is to have a strong idea about what you’re trying to achieve, and in that context review your text to see what exactly might help.\nTake a moment to look at the text. What do you notice?\nHere’s what I see:\nIt’s plain text so there is no markup to parse (yay!).\nThe translation of the original German uses UK English (e.g. “\ntravelling\n“).\nThe lines are artificially wrapped with new lines at about 70 characters (meh).\nThere are no obvious typos or spelling mistakes.\nThere’s punctuation like commas, apostrophes, quotes, question marks, and more.\nThere’s hyphenated descriptions like “armour-like”.\nThere’s a lot of use of the em dash (“-“) to continue sentences (maybe replace with commas?).\nThere are names (e.g. “\nMr. Samsa\n“)\nThere does not appear to be numbers that require handling (e.g. 1999)\nThere are section markers (e.g. “II” and “III”), and we have removed the first “I”.\nI’m sure there is a lot more going on to the trained eye.\nWe are going to look at general text cleaning steps in this tutorial.\nNevertheless, consider some possible objectives we may have when working with this text document.\nFor example:\nIf we were interested in developing a\nKafkaesque\nlanguage model, we may want to keep all of the case, quotes, and other punctuation in place.\nIf we were interested in classifying documents as “\nKafka\n” and “\nNot Kafka\n,” maybe we would want to strip case, punctuation, and even trim words back to their stem.\nUse your task as the lens by which to choose how to ready your text data.\nManual Tokenization\nText cleaning is hard, but the text we have chosen to work with is pretty clean already.\nWe could just write some Python code to clean it up manually, and this is a good exercise for those simple problems that you encounter. Tools like regular expressions and splitting strings can get you a long way.\n1. Load Data\nLet’s load the text data so that we can work with it.\nThe text is small and will load quickly and easily fit into memory. This will not always be the case and you may need to write code to memory map the file. Tools like NLTK (covered in the next section) will make working with large files much easier.\nWe can load the entire “\nmetamorphosis_clean.txt\n” into memory as follows:\n# load text\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n1\n2\n3\n4\n5\n# load text\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\nRunning the example loads the whole file into memory ready to work with.\n2. Split by Whitespace\nClean text often means a list of words or tokens that we can work with in our machine learning models.\nThis means converting the raw text into a list of words and saving it again.\nA very simple way to do this would be to split the document by white space, including ” “, new lines, tabs and more. We can do this in Python with the split() function on the loaded string.\n# load text\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into words by white space\nwords = text.split()\nprint(words[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n# load text\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into words by white space\nwords\n=\ntext\n.\nsplit\n(\n)\nprint\n(\nwords\n[\n:\n100\n]\n)\nRunning the example splits the document into a long list of words and prints the first 100 for us to review.\nWe can see that punctuation is preserved (e.g. “\nwasn’t\n” and “\narmour-like\n“), which is nice. We can also see that end of sentence punctuation is kept with the last word (e.g. “\nthought\n.”), which is not great.\n['One', 'morning,', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams,', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin.', 'He', 'lay', 'on', 'his', 'armour-like', 'back,', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly,', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections.', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment.', 'His', 'many', 'legs,', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him,', 'waved', 'about', 'helplessly', 'as', 'he', 'looked.', '\"What\\'s', 'happened', 'to', 'me?\"', 'he', 'thought.', 'It', \"wasn't\", 'a', 'dream.', 'His', 'room,', 'a', 'proper', 'human']\n1\n['One', 'morning,', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams,', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin.', 'He', 'lay', 'on', 'his', 'armour-like', 'back,', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly,', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections.', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment.', 'His', 'many', 'legs,', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him,', 'waved', 'about', 'helplessly', 'as', 'he', 'looked.', '\"What\\'s', 'happened', 'to', 'me?\"', 'he', 'thought.', 'It', \"wasn't\", 'a', 'dream.', 'His', 'room,', 'a', 'proper', 'human']\n3. Select Words\nAnother approach might be to use the regex model (re) and split the document into words by selecting for strings of alphanumeric characters (a-z, A-Z, 0-9 and ‘_’).\nFor example:\n# load text\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split based on words only\nimport re\nwords = re.split(r'\\W+', text)\nprint(words[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n# load text\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split based on words only\nimport\nre\nwords\n=\nre\n.\nsplit\n(\nr\n'\\W+'\n,\ntext\n)\nprint\n(\nwords\n[\n:\n100\n]\n)\nAgain, running the example we can see that we get our list of words. This time, we can see that “\narmour-like\n” is now two words “\narmour\n” and “\nlike\n” (fine) but contractions like “\nWhat’s\n” is also two words “\nWhat\n” and “\ns\n” (not great).\n['One', 'morning', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', 'He', 'lay', 'on', 'his', 'armour', 'like', 'back', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', 'His', 'many', 'legs', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', 'What', 's', 'happened', 'to', 'me', 'he', 'thought', 'It', 'wasn', 't', 'a', 'dream', 'His', 'room']\n1\n['One', 'morning', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', 'He', 'lay', 'on', 'his', 'armour', 'like', 'back', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', 'His', 'many', 'legs', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', 'What', 's', 'happened', 'to', 'me', 'he', 'thought', 'It', 'wasn', 't', 'a', 'dream', 'His', 'room']\n3. Split by Whitespace and Remove Punctuation\nNote: This example was written for Python 3.\nWe may want the words, but without the punctuation like commas and quotes. We also want to keep contractions together.\nOne way would be to split the document into words by white space (as in “\n2. Split by Whitespace\n“), then use string translation to replace all punctuation with nothing (e.g. remove it).\nPython provides a constant called\nstring.punctuation\nthat provides a great list of punctuation characters. For example:\nprint(string.punctuation)\n1\nprint\n(\nstring\n.\npunctuation\n)\nResults in:\n!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n1\n!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\nPython offers a function called\ntranslate()\nthat will map one set of characters to another.\nWe can use the function\nmaketrans()\nto create a mapping table. We can create an empty mapping table, but the third argument of this function allows us to list all of the characters to remove during the translation process. For example:\ntable = str.maketrans('', '', string.punctuation)\n1\ntable\n=\nstr\n.\nmaketrans\n(\n''\n,\n''\n,\nstring\n.\npunctuation\n)\nWe can put all of this together, load the text file, split it into words by white space, then translate each word to remove the punctuation.\n# load text\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into words by white space\nwords = text.split()\n# remove punctuation from each word\nimport string\ntable = str.maketrans('', '', string.punctuation)\nstripped = [w.translate(table) for w in words]\nprint(stripped[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n# load text\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into words by white space\nwords\n=\ntext\n.\nsplit\n(\n)\n# remove punctuation from each word\nimport\nstring\ntable\n=\nstr\n.\nmaketrans\n(\n''\n,\n''\n,\nstring\n.\npunctuation\n)\nstripped\n=\n[\nw\n.\ntranslate\n(\ntable\n)\nfor\nw\nin\nwords\n]\nprint\n(\nstripped\n[\n:\n100\n]\n)\nWe can see that this has had the desired effect, mostly.\nContractions like “\nWhat’s\n” have become “\nWhats\n” but “\narmour-like\n” has become “\narmourlike\n“.\n['One', 'morning', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', 'He', 'lay', 'on', 'his', 'armourlike', 'back', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', 'His', 'many', 'legs', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', 'Whats', 'happened', 'to', 'me', 'he', 'thought', 'It', 'wasnt', 'a', 'dream', 'His', 'room', 'a', 'proper', 'human']\n1\n['One', 'morning', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', 'He', 'lay', 'on', 'his', 'armourlike', 'back', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', 'His', 'many', 'legs', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', 'Whats', 'happened', 'to', 'me', 'he', 'thought', 'It', 'wasnt', 'a', 'dream', 'His', 'room', 'a', 'proper', 'human']\nIf you know anything about regex, then you know things can get complex from here.\n4. Normalizing Case\nIt is common to convert all words to one case.\nThis means that the vocabulary will shrink in size, but some distinctions are lost (e.g. “\nApple\n” the company vs “\napple\n” the fruit is a commonly used example).\nWe can convert all words to lowercase by calling the lower() function on each word.\nFor example:\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into words by white space\nwords = text.split()\n# convert to lower case\nwords = [word.lower() for word in words]\nprint(words[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n9\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into words by white space\nwords\n=\ntext\n.\nsplit\n(\n)\n# convert to lower case\nwords\n=\n[\nword\n.\nlower\n(\n)\nfor\nword\nin\nwords\n]\nprint\n(\nwords\n[\n:\n100\n]\n)\nRunning the example, we can see that all words are now lowercase.\n['one', 'morning,', 'when', 'gregor', 'samsa', 'woke', 'from', 'troubled', 'dreams,', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin.', 'he', 'lay', 'on', 'his', 'armour-like', 'back,', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly,', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections.', 'the', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment.', 'his', 'many', 'legs,', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him,', 'waved', 'about', 'helplessly', 'as', 'he', 'looked.', '\"what\\'s', 'happened', 'to', 'me?\"', 'he', 'thought.', 'it', \"wasn't\", 'a', 'dream.', 'his', 'room,', 'a', 'proper', 'human']\n1\n['one', 'morning,', 'when', 'gregor', 'samsa', 'woke', 'from', 'troubled', 'dreams,', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin.', 'he', 'lay', 'on', 'his', 'armour-like', 'back,', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly,', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections.', 'the', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment.', 'his', 'many', 'legs,', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him,', 'waved', 'about', 'helplessly', 'as', 'he', 'looked.', '\"what\\'s', 'happened', 'to', 'me?\"', 'he', 'thought.', 'it', \"wasn't\", 'a', 'dream.', 'his', 'room,', 'a', 'proper', 'human']\nNote\nCleaning text is really hard, problem specific, and full of tradeoffs.\nRemember, simple is better.\nSimpler text data, simpler models, smaller vocabularies. You can always make things more complex later to see if it results in better model skill.\nNext, we’ll look at some of the tools in the NLTK library that offer more than simple string splitting.\nTokenization and Cleaning with NLTK\nThe\nNatural Language Toolkit\n, or NLTK for short, is a Python library written for working and modeling text.\nIt provides good tools for loading and cleaning text that we can use to get our data ready for working with machine learning and deep learning algorithms.\n1. Install NLTK\nYou can install NLTK using your favorite package manager, such as pip:\nsudo pip install -U nltk\n1\nsudo pip install -U nltk\nAfter installation, you will need to install the data used with the library, including a great set of documents that you can use later for testing other tools in NLTK.\nThere are few ways to do this, such as from within a script:\nimport nltk\nnltk.download()\n1\n2\nimport\nnltk\nnltk\n.\ndownload\n(\n)\nOr from the command line:\npython -m nltk.downloader all\n1\npython -m nltk.downloader all\nFor more help installing and setting up NLTK, see:\nInstalling NLTK\nInstalling NLTK Data\n2. Split into Sentences\nA good useful first step is to split the text into sentences.\nSome modeling tasks prefer input to be in the form of paragraphs or sentences, such as word2vec. You could first split your text into sentences, split each sentence into words, then save each sentence to file, one per line.\nNLTK provides the\nsent_tokenize()\nfunction to split text into sentences.\nThe example below loads the “\nmetamorphosis_clean.txt\n” file into memory, splits it into sentences, and prints the first sentence.\n# load data\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into sentences\nfrom nltk import sent_tokenize\nsentences = sent_tokenize(text)\nprint(sentences[0])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n# load data\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into sentences\nfrom\nnltk\nimport\nsent_tokenize\nsentences\n=\nsent_tokenize\n(\ntext\n)\nprint\n(\nsentences\n[\n0\n]\n)\nRunning the example, we can see that although the document is split into sentences, that each sentence still preserves the new line from the artificial wrap of the lines in the original document.\nOne morning, when Gregor Samsa woke from troubled dreams, he found\nhimself transformed in his bed into a horrible vermin.\n3. Split into Words\nNLTK provides a function called\nword_tokenize()\nfor splitting strings into tokens (nominally words).\nIt splits tokens based on white space and punctuation. For example, commas and periods are taken as separate tokens. Contractions are split apart (e.g. “\nWhat’s\n” becomes “\nWhat\n” “‘\ns\n“). Quotes are kept, and so on.\nFor example:\n# load data\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into words\nfrom nltk.tokenize import word_tokenize\ntokens = word_tokenize(text)\nprint(tokens[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n# load data\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into words\nfrom\nnltk\n.\ntokenize\nimport\nword_tokenize\ntokens\n=\nword_tokenize\n(\ntext\n)\nprint\n(\ntokens\n[\n:\n100\n]\n)\nRunning the code, we can see that punctuation are now tokens that we could then decide to specifically filter out.\n['One', 'morning', ',', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', ',', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', '.', 'He', 'lay', 'on', 'his', 'armour-like', 'back', ',', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', ',', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', '.', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', '.', 'His', 'many', 'legs', ',', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', ',', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', '.', '``', 'What', \"'s\", 'happened', 'to']\n1\n['One', 'morning', ',', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', ',', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', '.', 'He', 'lay', 'on', 'his', 'armour-like', 'back', ',', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', ',', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', '.', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', '.', 'His', 'many', 'legs', ',', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', ',', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', '.', '``', 'What', \"'s\", 'happened', 'to']\n4. Filter Out Punctuation\nWe can filter out all tokens that we are not interested in, such as all standalone punctuation.\nThis can be done by iterating over all tokens and only keeping those tokens that are all alphabetic. Python has the function\nisalpha()\nthat can be used. For example:\n# load data\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into words\nfrom nltk.tokenize import word_tokenize\ntokens = word_tokenize(text)\n# remove all tokens that are not alphabetic\nwords = [word for word in tokens if word.isalpha()]\nprint(words[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n# load data\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into words\nfrom\nnltk\n.\ntokenize\nimport\nword_tokenize\ntokens\n=\nword_tokenize\n(\ntext\n)\n# remove all tokens that are not alphabetic\nwords\n=\n[\nword\nfor\nword\nin\ntokens\nif\nword\n.\nisalpha\n(\n)\n]\nprint\n(\nwords\n[\n:\n100\n]\n)\nRunning the example, you can see that not only punctuation tokens, but examples like “\narmour-like\n” and “\n‘s\n” were also filtered out.\n['One', 'morning', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', 'He', 'lay', 'on', 'his', 'back', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', 'His', 'many', 'legs', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', 'What', 'happened', 'to', 'me', 'he', 'thought', 'It', 'was', 'a', 'dream', 'His', 'room', 'a', 'proper', 'human', 'room']\n1\n['One', 'morning', 'when', 'Gregor', 'Samsa', 'woke', 'from', 'troubled', 'dreams', 'he', 'found', 'himself', 'transformed', 'in', 'his', 'bed', 'into', 'a', 'horrible', 'vermin', 'He', 'lay', 'on', 'his', 'back', 'and', 'if', 'he', 'lifted', 'his', 'head', 'a', 'little', 'he', 'could', 'see', 'his', 'brown', 'belly', 'slightly', 'domed', 'and', 'divided', 'by', 'arches', 'into', 'stiff', 'sections', 'The', 'bedding', 'was', 'hardly', 'able', 'to', 'cover', 'it', 'and', 'seemed', 'ready', 'to', 'slide', 'off', 'any', 'moment', 'His', 'many', 'legs', 'pitifully', 'thin', 'compared', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', 'waved', 'about', 'helplessly', 'as', 'he', 'looked', 'What', 'happened', 'to', 'me', 'he', 'thought', 'It', 'was', 'a', 'dream', 'His', 'room', 'a', 'proper', 'human', 'room']\n5. Filter out Stop Words (and Pipeline)\nStop words\nare those words that do not contribute to the deeper meaning of the phrase.\nThey are the most common words such as: “\nthe\n“, “\na\n“, and “\nis\n“.\nFor some applications like documentation classification, it may make sense to remove stop words.\nNLTK provides a list of commonly agreed upon stop words for a variety of languages, such as English. They can be loaded as follows:\nfrom nltk.corpus import stopwords\nstop_words = stopwords.words('english')\nprint(stop_words)\n1\n2\n3\nfrom\nnltk\n.\ncorpus\nimport\nstopwords\nstop_words\n=\nstopwords\n.\nwords\n(\n'english'\n)\nprint\n(\nstop_words\n)\nYou can see the full list as follows:\n['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers', 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', 'couldn', 'didn', 'doesn', 'hadn', 'hasn', 'haven', 'isn', 'ma', 'mightn', 'mustn', 'needn', 'shan', 'shouldn', 'wasn', 'weren', 'won', 'wouldn']\n1\n['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers', 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', 'couldn', 'didn', 'doesn', 'hadn', 'hasn', 'haven', 'isn', 'ma', 'mightn', 'mustn', 'needn', 'shan', 'shouldn', 'wasn', 'weren', 'won', 'wouldn']\nYou can see that they are all lower case and have punctuation removed.\nYou could compare your tokens to the stop words and filter them out, but you must ensure that your text is prepared the same way.\nLet’s demonstrate this with a small pipeline of text preparation including:\nLoad the raw text.\nSplit into tokens.\nConvert to lowercase.\nRemove punctuation from each token.\nFilter out remaining tokens that are not alphabetic.\nFilter out tokens that are stop words.\n# load data\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into words\nfrom nltk.tokenize import word_tokenize\ntokens = word_tokenize(text)\n# convert to lower case\ntokens = [w.lower() for w in tokens]\n# remove punctuation from each word\nimport string\ntable = str.maketrans('', '', string.punctuation)\nstripped = [w.translate(table) for w in tokens]\n# remove remaining tokens that are not alphabetic\nwords = [word for word in stripped if word.isalpha()]\n# filter out stop words\nfrom nltk.corpus import stopwords\nstop_words = set(stopwords.words('english'))\nwords = [w for w in words if not w in stop_words]\nprint(words[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n# load data\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into words\nfrom\nnltk\n.\ntokenize\nimport\nword_tokenize\ntokens\n=\nword_tokenize\n(\ntext\n)\n# convert to lower case\ntokens\n=\n[\nw\n.\nlower\n(\n)\nfor\nw\nin\ntokens\n]\n# remove punctuation from each word\nimport\nstring\ntable\n=\nstr\n.\nmaketrans\n(\n''\n,\n''\n,\nstring\n.\npunctuation\n)\nstripped\n=\n[\nw\n.\ntranslate\n(\ntable\n)\nfor\nw\nin\ntokens\n]\n# remove remaining tokens that are not alphabetic\nwords\n=\n[\nword\nfor\nword\nin\nstripped\nif\nword\n.\nisalpha\n(\n)\n]\n# filter out stop words\nfrom\nnltk\n.\ncorpus\nimport\nstopwords\nstop_words\n=\nset\n(\nstopwords\n.\nwords\n(\n'english'\n)\n)\nwords\n=\n[\nw\nfor\nw\nin\nwords\nif\nnot\nw\nin\nstop_words\n]\nprint\n(\nwords\n[\n:\n100\n]\n)\nRunning this example, we can see that in addition to all of the other transforms, stop words like “\na\n” and “\nto\n” have been removed.\nI note that we are still left with tokens like “\nnt\n“. The rabbit hole is deep; there’s always more we can do.\n['one', 'morning', 'gregor', 'samsa', 'woke', 'troubled', 'dreams', 'found', 'transformed', 'bed', 'horrible', 'vermin', 'lay', 'armourlike', 'back', 'lifted', 'head', 'little', 'could', 'see', 'brown', 'belly', 'slightly', 'domed', 'divided', 'arches', 'stiff', 'sections', 'bedding', 'hardly', 'able', 'cover', 'seemed', 'ready', 'slide', 'moment', 'many', 'legs', 'pitifully', 'thin', 'compared', 'size', 'rest', 'waved', 'helplessly', 'looked', 'happened', 'thought', 'nt', 'dream', 'room', 'proper', 'human', 'room', 'although', 'little', 'small', 'lay', 'peacefully', 'four', 'familiar', 'walls', 'collection', 'textile', 'samples', 'lay', 'spread', 'table', 'samsa', 'travelling', 'salesman', 'hung', 'picture', 'recently', 'cut', 'illustrated', 'magazine', 'housed', 'nice', 'gilded', 'frame', 'showed', 'lady', 'fitted', 'fur', 'hat', 'fur', 'boa', 'sat', 'upright', 'raising', 'heavy', 'fur', 'muff', 'covered', 'whole', 'lower', 'arm', 'towards', 'viewer']\n1\n['one', 'morning', 'gregor', 'samsa', 'woke', 'troubled', 'dreams', 'found', 'transformed', 'bed', 'horrible', 'vermin', 'lay', 'armourlike', 'back', 'lifted', 'head', 'little', 'could', 'see', 'brown', 'belly', 'slightly', 'domed', 'divided', 'arches', 'stiff', 'sections', 'bedding', 'hardly', 'able', 'cover', 'seemed', 'ready', 'slide', 'moment', 'many', 'legs', 'pitifully', 'thin', 'compared', 'size', 'rest', 'waved', 'helplessly', 'looked', 'happened', 'thought', 'nt', 'dream', 'room', 'proper', 'human', 'room', 'although', 'little', 'small', 'lay', 'peacefully', 'four', 'familiar', 'walls', 'collection', 'textile', 'samples', 'lay', 'spread', 'table', 'samsa', 'travelling', 'salesman', 'hung', 'picture', 'recently', 'cut', 'illustrated', 'magazine', 'housed', 'nice', 'gilded', 'frame', 'showed', 'lady', 'fitted', 'fur', 'hat', 'fur', 'boa', 'sat', 'upright', 'raising', 'heavy', 'fur', 'muff', 'covered', 'whole', 'lower', 'arm', 'towards', 'viewer']\n6. Stem Words\nStemming\nrefers to the process of reducing each word to its root or base.\nFor example “\nfishing\n,” “\nfished\n,” “\nfisher\n” all reduce to the stem “\nfish\n.”\nSome applications, like document classification, may benefit from stemming in order to both reduce the vocabulary and to focus on the sense or sentiment of a document rather than deeper meaning.\nThere are many stemming algorithms, although a popular and long-standing method is the Porter Stemming algorithm. This method is available in NLTK via the\nPorterStemmer\nclass.\nFor example:\n# load data\nfilename = 'metamorphosis_clean.txt'\nfile = open(filename, 'rt')\ntext = file.read()\nfile.close()\n# split into words\nfrom nltk.tokenize import word_tokenize\ntokens = word_tokenize(text)\n# stemming of words\nfrom nltk.stem.porter import PorterStemmer\nporter = PorterStemmer()\nstemmed = [porter.stem(word) for word in tokens]\nprint(stemmed[:100])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n# load data\nfilename\n=\n'metamorphosis_clean.txt'\nfile\n=\nopen\n(\nfilename\n,\n'rt'\n)\ntext\n=\nfile\n.\nread\n(\n)\nfile\n.\nclose\n(\n)\n# split into words\nfrom\nnltk\n.\ntokenize\nimport\nword_tokenize\ntokens\n=\nword_tokenize\n(\ntext\n)\n# stemming of words\nfrom\nnltk\n.\nstem\n.\nporter\nimport\nPorterStemmer\nporter\n=\nPorterStemmer\n(\n)\nstemmed\n=\n[\nporter\n.\nstem\n(\nword\n)\nfor\nword\nin\ntokens\n]\nprint\n(\nstemmed\n[\n:\n100\n]\n)\nRunning the example, you can see that words have been reduced to their stems, such as “\ntrouble\n” has become “\ntroubl\n“. You can also see that the stemming implementation has also reduced the tokens to lowercase, likely for internal look-ups in word tables.\nYou can also see that the stemming implementation has also reduced the tokens to lowercase, likely for internal look-ups in word tables.\n['one', 'morn', ',', 'when', 'gregor', 'samsa', 'woke', 'from', 'troubl', 'dream', ',', 'he', 'found', 'himself', 'transform', 'in', 'hi', 'bed', 'into', 'a', 'horribl', 'vermin', '.', 'He', 'lay', 'on', 'hi', 'armour-lik', 'back', ',', 'and', 'if', 'he', 'lift', 'hi', 'head', 'a', 'littl', 'he', 'could', 'see', 'hi', 'brown', 'belli', ',', 'slightli', 'dome', 'and', 'divid', 'by', 'arch', 'into', 'stiff', 'section', '.', 'the', 'bed', 'wa', 'hardli', 'abl', 'to', 'cover', 'it', 'and', 'seem', 'readi', 'to', 'slide', 'off', 'ani', 'moment', '.', 'hi', 'mani', 'leg', ',', 'piti', 'thin', 'compar', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', ',', 'wave', 'about', 'helplessli', 'as', 'he', 'look', '.', '``', 'what', \"'s\", 'happen', 'to'\n1\n['one', 'morn', ',', 'when', 'gregor', 'samsa', 'woke', 'from', 'troubl', 'dream', ',', 'he', 'found', 'himself', 'transform', 'in', 'hi', 'bed', 'into', 'a', 'horribl', 'vermin', '.', 'He', 'lay', 'on', 'hi', 'armour-lik', 'back', ',', 'and', 'if', 'he', 'lift', 'hi', 'head', 'a', 'littl', 'he', 'could', 'see', 'hi', 'brown', 'belli', ',', 'slightli', 'dome', 'and', 'divid', 'by', 'arch', 'into', 'stiff', 'section', '.', 'the', 'bed', 'wa', 'hardli', 'abl', 'to', 'cover', 'it', 'and', 'seem', 'readi', 'to', 'slide', 'off', 'ani', 'moment', '.', 'hi', 'mani', 'leg', ',', 'piti', 'thin', 'compar', 'with', 'the', 'size', 'of', 'the', 'rest', 'of', 'him', ',', 'wave', 'about', 'helplessli', 'as', 'he', 'look', '.', '``', 'what', \"'s\", 'happen', 'to'\nThere is a nice suite of stemming and lemmatization algorithms to choose from in NLTK, if reducing words to their root is something you need for your project.\nAdditional Text Cleaning Considerations\nWe are only getting started.\nBecause the source text for this tutorial was reasonably clean to begin with, we skipped many concerns of text cleaning that you may need to deal with in your own project.\nHere is a short list of additional considerations when cleaning text:\nHandling large documents and large collections of text documents that do not fit into memory.\nExtracting text from markup like HTML, PDF, or other structured document formats.\nTransliteration of characters from other languages into English.\nDecoding Unicode characters into a normalized form, such as UTF8.\nHandling of domain specific words, phrases, and acronyms.\nHandling or removing numbers, such as dates and amounts.\nLocating and correcting common typos and misspellings.\n…\nThe list could go on.\nHopefully, you can see that getting truly clean text is impossible, that we are really doing the best we can based on the time, resources, and knowledge we have.\nThe idea of “clean” is really defined by the specific task or concern of your project.\nA pro tip is to continually review your tokens after every transform. I have tried to show that in this tutorial and I hope you take that to heart.\nIdeally, you would save a new file after each transform so that you can spend time with all of the data in the new form. Things always jump out at you when to take the time to review your data.\nHave you done some text cleaning before? What are you preferred pipeline of transforms?\nLet me know in the comments below.\nTips for Cleaning Text for Word Embedding\nRecently, the field of natural language processing has been moving away from bag-of-word models and word encoding toward word embeddings.\nThe benefit of word embeddings is that they encode each word into a dense vector that captures something about its relative meaning within the training text.\nThis means that variations of words like case, spelling, punctuation, and so on will automatically be learned to be similar in the embedding space. In turn, this can mean that the amount of cleaning required from your text may be less and perhaps quite different to classical text cleaning.\nFor example, it may no-longer make sense to stem words or remove punctuation for contractions.\nTomas Mikolov is one of the developers of word2vec, a popular word embedding method. He suggests only very minimal text cleaning is required when learning a word embedding model.\nBelow is his response when pressed with the question about how to best prepare text data for word2vec.\nThere is no universal answer. It all depends on what you plan to use the vectors for. In my experience, it is usually good to disconnect (or remove) punctuation from words, and sometimes also convert all characters to lowercase. One can also replace all numbers (possibly greater than some constant) with some single token such as .\nAll these pre-processing steps aim to reduce the vocabulary size without removing any important content (which in some cases may not be true when you lowercase certain words, ie. ‘Bush’ is different than ‘bush’, while ‘Another’ has usually the same sense as ‘another’). The smaller the vocabulary is, the lower is the memory complexity, and the more robustly are the parameters for the words estimated. You also have to pre-process the test data in the same way.\n…\nIn short, you will understand all this much better if you will run experiments.\nRead the full thread on Google Groups\n.\nFurther Reading\nThis section provides more resources on the topic if you are looking go deeper.\nMetamorphosis by Franz Kafka on Project Gutenberg\nnltk.tokenize package API\nnltk.stem package API\nChapter 3: Processing Raw Text, Natural Language Processing with Python\nSummary\nIn this tutorial, you discovered how to clean text or machine learning in Python.\nSpecifically, you learned:\nHow to get started by developing your own very simple text cleaning tools.\nHow to take a step up and use the more sophisticated methods in the NLTK library.\nHow to prepare text when using modern text representation methods like word embeddings.\nDo you have any questions?\nAsk your questions in the comments below and I will do my best to answer.\nDo you have experience with cleaning text?\nPlease share your experiences in the comments below.\nDevelop Deep Learning models for Text Data Today!\nDevelop Your Own Text models in Minutes\n...with just a few lines of python code\nDiscover how in my new Ebook:\nDeep Learning for Natural Language Processing\nIt provides\nself-study tutorials\non topics like:\nBag-of-Words, Word Embedding, Language Models, Caption Generation, Text Translation\nand much more...\nFinally Bring Deep Learning to your Natural Language Processing Projects\nSkip the Academics. Just Results.\nSee What's Inside\nShare\nPost\nShare\nMore On This Topic\nHow to Encode Text Data for Machine Learning with…\nA Gentle Introduction to Calculating the BLEU Score…\nText Generation With LSTM Recurrent Neural Networks…\nEncoder-Decoder Deep Learning Models for Text Summarization\nBest Practices for Text Classification with Deep Learning\nHow to Develop a Deep Learning Bag-of-Words Model…\nAbout Jason Brownlee\nJason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with modern machine learning methods via hands-on tutorials.\nView all posts by Jason Brownlee\n→",
  "metadata": {
    "title": "How to Clean Text for Machine Learning with Python - MachineLearningMastery.com",
    "description": null,
    "keywords": null,
    "author": "Jason Brownlee",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "How to Clean Text for Machine Learning with Python - MachineLearningMastery.com",
    "og:description": "You cannot go straight from raw text to fitting a machine learning or deep learning model. You must clean your text first, which means splitting it into words and handling punctuation and case. In fact, there is a whole suite of text preparation methods that you may need to use, and the choice of methods really depends on your natural…",
    "og:url": "https://machinelearningmastery.com/clean-text-machine-learning-python/",
    "og:site_name": "MachineLearningMastery.com",
    "og:image": "https://machinelearningmastery.com/wp-content/uploads/2018/11/How-to-Develop-Multilayer-Perceptron-Models-for-Time-Series-Forecasting.jpg",
    "og:image:width": "640",
    "og:image:height": "461",
    "og:image:type": "image/jpeg",
    "twitter:label1": "Written by",
    "twitter:data1": "Jason Brownlee",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "22 minutes",
    "twitter:card": "summary_large_image",
    "twitter:description": "You cannot go straight from raw text to fitting a machine learning or deep learning model. You must clean your text first, which means splitting it into words and handling punctuation and case. In fact, there is a whole suite of text preparation methods that you may need to use, and the choice of methods really depends on your natural…",
    "twitter:title": "How to Clean Text for Machine Learning with Python - MachineLearningMastery.com",
    "article:publisher": "https://www.facebook.com/MachineLearningMastery/",
    "article:author": "https://www.facebook.com/MachineLearningMastery/",
    "article:published_time": "2017-10-17T18:00:56+00:00",
    "article:modified_time": "2019-08-07T04:39:46+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2018/11/How-to-Develop-Multilayer-Perceptron-Models-for-Time-Series-Forecasting.jpg",
      "bytes": 49106,
      "reused": true,
      "local_path": "attachments/59767026df5a/images/000.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2018/11/How-to-Develop-Multilayer-Perceptron-Models-for-Time-Series-Forecasting.jpg",
      "alt": "How to Develop Multilayer Perceptron Models for Time Series Forecasting",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4NLP-220.png",
      "bytes": 4110,
      "reused": false,
      "local_path": "attachments/59767026df5a/images/body-e8fcb21cfc1be3513af1.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2022/11/DL4NLP-220.png",
      "alt": "Deep Learning for Natural Language Processing",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/09/How-to-Prepare-Text-Data-for-Machine-Learning-with-scikit-learn.jpg",
      "bytes": 56900,
      "reused": true,
      "local_path": "attachments/59767026df5a/images/003.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/09/How-to-Prepare-Text-Data-for-Machine-Learning-with-scikit-learn.jpg",
      "alt": "How to Prepare Text Data for Machine Learning with scikit-learn",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/11/A-Gentle-Introduction-to-Calculating-the-BLEU-Score-for-Text-in-Python.jpg",
      "bytes": 98008,
      "reused": true,
      "local_path": "attachments/59767026df5a/images/005.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/11/A-Gentle-Introduction-to-Calculating-the-BLEU-Score-for-Text-in-Python.jpg",
      "alt": "A Gentle Introduction to Calculating the BLEU Score for Text in Python",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2016/08/Text-Generation-With-LSTM-Recurrent-Neural-Networks-in-Python-with-Keras.jpg",
      "bytes": 30901,
      "reused": true,
      "local_path": "attachments/59767026df5a/images/007.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2016/08/Text-Generation-With-LSTM-Recurrent-Neural-Networks-in-Python-with-Keras.jpg",
      "alt": "Text Generation With LSTM Recurrent Neural Networks in Python with Keras",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/12/Encoder-Decoder-Deep-Learning-Models-for-Text-Summarization.jpg",
      "bytes": 41860,
      "reused": true,
      "local_path": "attachments/59767026df5a/images/009.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/12/Encoder-Decoder-Deep-Learning-Models-for-Text-Summarization.jpg",
      "alt": "Encoder-Decoder Deep Learning Models for Text Summarization",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/10/Best-Practices-for-Document-Classification-with-Deep-Learning.jpg",
      "bytes": 57175,
      "reused": true,
      "local_path": "attachments/59767026df5a/images/011.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/10/Best-Practices-for-Document-Classification-with-Deep-Learning.jpg",
      "alt": "Best Practices for Document Classification with Deep Learning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2017/10/How-to-Develop-a-Deep-Learning-Bag-of-Words-Model-for-Predicting-Sentiment-in-Movie-Reviews.jpg",
      "bytes": 54545,
      "reused": true,
      "local_path": "attachments/59767026df5a/images/013.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2017/10/How-to-Develop-a-Deep-Learning-Bag-of-Words-Model-for-Predicting-Sentiment-in-Movie-Reviews.jpg",
      "alt": "How to Develop a Deep Learning Bag-of-Words Model for Predicting Sentiment in Movie Reviews",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://machinelearningmastery.com/author/jasonb/",
      "label": "Jason Brownlee",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/category/natural-language-processing/",
      "label": "Deep Learning for Natural Language Processing",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/clean-text-machine-learning-python/",
      "label": "146",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/deep-learning-for-nlp/",
      "label": "Deep Learning for Natural Language Processing",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/The_Metamorphosis",
      "label": "Metamorphosis",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Franz_Kafka",
      "label": "Franz Kafka",
      "type": "a"
    },
    {
      "href": "http://www.gutenberg.org/ebooks/5200",
      "label": "Metamorphosis by Franz Kafka on Project Gutenberg",
      "type": "a"
    },
    {
      "href": "http://www.gutenberg.org/cache/epub/5200/pg5200.txt",
      "label": "Metamorphosis by Franz Kafka Plain Text UTF-8",
      "type": "a"
    },
    {
      "href": "http://www.thefreedictionary.com/Kafkaesk",
      "label": "Kafkaesque",
      "type": "a"
    },
    {
      "href": "https://docs.python.org/3/library/stdtypes.html",
      "label": "translate()",
      "type": "a"
    },
    {
      "href": "http://www.nltk.org/",
      "label": "Natural Language Toolkit",
      "type": "a"
    },
    {
      "href": "http://www.nltk.org/install.html",
      "label": "Installing NLTK",
      "type": "a"
    },
    {
      "href": "http://www.nltk.org/data.html",
      "label": "Installing NLTK Data",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Stop_words",
      "label": "Stop words",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Stemming",
      "label": "Stemming",
      "type": "a"
    },
    {
      "href": "https://tartarus.org/martin/PorterStemmer/",
      "label": "PorterStemmer",
      "type": "a"
    },
    {
      "href": "https://groups.google.com/d/msg/word2vec-toolkit/jPfyP6FoB94/tGzZxScO0GsJ",
      "label": "Read the full thread on Google Groups",
      "type": "a"
    },
    {
      "href": "http://www.nltk.org/api/nltk.tokenize.html",
      "label": "nltk.tokenize package API",
      "type": "a"
    },
    {
      "href": "http://www.nltk.org/api/nltk.stem.html",
      "label": "nltk.stem package API",
      "type": "a"
    },
    {
      "href": "http://www.nltk.org/book/ch03.html",
      "label": "Chapter 3: Processing Raw Text, Natural Language Processing with Python",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/prepare-text-data-machine-learning-scikit-learn/",
      "label": "How to Encode Text Data for Machine Learning with…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/calculate-bleu-score-for-text-python/",
      "label": "A Gentle Introduction to Calculating the BLEU Score…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/",
      "label": "Text Generation With LSTM Recurrent Neural Networks…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/encoder-decoder-deep-learning-models-text-summarization/",
      "label": "Encoder-Decoder Deep Learning Models for Text Summarization",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/best-practices-document-classification-deep-learning/",
      "label": "Best Practices for Text Classification with Deep Learning",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/deep-learning-bag-of-words-model-sentiment-analysis/",
      "label": "How to Develop a Deep Learning Bag-of-Words Model…",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:20Z",
  "source_url": "https://machinelearningmastery.com/clean-text-machine-learning-python/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 41389,
  "expected_body_images": 8,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/59767026df5a/pages/source.html",
  "local_html": "attachments/59767026df5a/pages/article.html",
  "body_base_path": "attachments/59767026df5a/pages/",
  "body_markdown_path": "attachments/59767026df5a/pages/article.md"
}