{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "amananandrai",
  "title_listed": "14 tasks for text preprocessing in NLP",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/amananandrai/14-tasks-for-text-preprocessing-in-nlp-33n",
  "publish_date": "2021-03-20",
  "corpus_stage": "",
  "notes": "",
  "id": "8bc6c41ef74e",
  "collected_at": "2026-10-02T14:02:23",
  "title": "14 tasks for text preprocessing in NLP - DEV Community",
  "body_markdown": "Natural Language Processing a subfield of Machine Learning mainly deals with text data. It analyses reviews of objects like books, movies, play store apps, etc, to find whether they are positive or negative, sentiment analysis, text generation for chatbots, query analysis and resolution for search engines, and many other text-related tasks.\nPreprocessing of datasets is one of the most arduous tasks of the machine learning pipeline. Text preprocessing also requires many steps. Some of the tasks while dealing with text datasets is given below.\nLower casing\nAll the text data is converted into the lower case to make all the words with different casing get the same weightage.\nRemoval of punctuation\nAll the punctuation symbols are removed from the dataset as they are not important in many tasks for word prediction and sentiment analysis.\nRemoval of Stopwords\nStopwords are English words which does not add much meaning to a sentence. They can safely be ignored without sacrificing the meaning of the sentence. For example, the words like the, he, have, etc. These stopwords are removed from the dataset.\nRemoval of frequent words\nSometimes the frequent words are also removed to increase classification accuracy in text classification tasks because they are present in all the classes and removing them causes the accuracy to increase.\nRemoval of Rare words\nIn some of the cases, rare words are also ignored and therefore removed because they work as outliers.\nStemming\nStemming means to chop off the end of the words to make it similar to the root word like removing \"ing\", \"ant\" from \"consulting\" and \"consultant\"  to make it \"consult\".\nLemmatization\nLemmatization means to change the words to root words by the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma.\neg-\n1) am, are, is => be\n2) operating, operates, operation, operative, operatives, operational => operate\nIn the second example if stemming is performed then instead of \"operate\" the words will change to \"operat\" as it does not take into account the meaning of the words and just chop offs the characters from the last.\nRemoval of emojis\nIn today's world emojis are a must in text messages but they can be dealt in two ways the first way is to remove them from the dataset.\nRemoval of emoticons\nEmoticons are also removed from the dataset for many datasets.\nConversion of emoticons to words\nThe other way to deal with emoticons is to convert them to words.\nConversion of emojis to words\nEmojis can also be converted to relatable words.\nRemoval of URLs\nThe URLs present must be removed from the dataset.\nRemoval of HTML tags\nSometimes while scrapping data from websites HTML tags are included in the datasets which must be removed to make better language models.\nSpelling correction\nSpelling mistakes must be corrected to make better language mistakes. Minimum edit distance can be used to find words which are slightly altered from the original.",
  "metadata": {
    "title": "14 tasks for text preprocessing in NLP - DEV Community",
    "description": "Natural Language Processing a subfield of Machine Learning mainly deals with text data. It analyses r... Tagged with machinelearning, nlp.",
    "keywords": "machinelearning, nlp, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/amananandrai/14-tasks-for-text-preprocessing-in-nlp-33n",
    "og:title": "14 tasks for text preprocessing in NLP",
    "og:description": "Natural Language Processing a subfield of Machine Learning mainly deals with text data. It analyses r...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbcbkmj09mqobzrg7xj89.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "14 tasks for text preprocessing in NLP",
    "twitter:description": "Natural Language Processing a subfield of Machine Learning mainly deals with text data. It analyses r...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbcbkmj09mqobzrg7xj89.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:18Z",
  "source_url": "https://dev.to/amananandrai/14-tasks-for-text-preprocessing-in-nlp-33n",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 3056,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/8bc6c41ef74e/pages/source.html",
  "local_html": "attachments/8bc6c41ef74e/pages/article.html",
  "body_base_path": "attachments/8bc6c41ef74e/pages/",
  "body_markdown_path": "attachments/8bc6c41ef74e/pages/article.md"
}