{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "NLP101 — Getting Started with Natural Language Processing",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/natural-language-processing-101-4226",
  "publish_date": "2021-05-03",
  "corpus_stage": "",
  "notes": "",
  "id": "bd5215b0744e",
  "collected_at": "2026-10-02T14:01:33",
  "title": "NLP101 — Getting Started with Natural Language Processing - DEV Community",
  "body_markdown": "With the huge influx of unstructured text data from a plethora of social media platforms , different forums and a whole wealth of documents, it’s evident that processing these sources of data to distill the information that they contain is challenging because of the inherent complexity involved in processing them.\nNatural Language Processing (NLP) helps greatly in processing, analyzing and understanding these sources to gain information and meaningful insights; With the recent advances in computing and easier access to computing resources, certain Deep Learning models have achieved SOTA in solving some of the most challenging NLP tasks.\nThe NLP series by Women Who Code Data Science track gives the learners a comprehensive learning path; starting from the basics of NLP, gradually introducing advanced concepts like Deep Learning approaches to solve NLP tasks.\nIn this blog post, let us focus on answering the following questions.\nWhat is NLP?\nWhat are some interesting use cases of NLP?\nWhat are the challenges in processing natural language?\nWhat are the steps in a generic NLP task?\nWhat are common text pre-processing techniques?\nWhat is NLP?\nNatural language processing (NLP) can be considered to be a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language; in particular, how to program computers to process and analyze large amounts of natural language data.\nWith interesting applications such as text classification, sentiment analysis, machine translation, speech to text, text to speech, and so on, NLP has evolved over the past few decades from rule-based approaches, statistical techniques to AI-powered applications in the recent past.\nInteresting use cases of NLP\nLet’s take a look at some of the common use cases of NLP.\nMachine Translation\nMachine Translation is the task of automatically converting one natural language into another while preserving the meaning of the input text and producing fluent text in the output language. However, this task of machine translation comes with inherent challenges.\nText Classification\nText Classification is the process of assigning tags or categories to text according to its content.\nIt’s a fundamental problem in NLP and can be done either manually(tedious, time-consuming, and susceptible to human errors) or by leveraging ML techniques.\nSentiment Analysis\nSentiment Analysis is the contextual mining of text which identifies and extracts subjective information in the source text, such as recognizing polarity(positive, negative, neutral), identifying emotions, etc.\nA typical example is in the e-commerce industry, where mining and analyzing reviews for gaining insights on customer satisfaction and experience, identifying potential areas for improvement are important.\nVirtual assistants such as Siri, Alexa and Cortana; Google Translate, Speech to text and text to speech converters are all cool NLP applications that we use in our everyday lives!\nChallenges in understanding natural language\nNatural language has such great diversity, and every language has its own rich grammar and uniqueness. The following are some of the inherent challenges that arise in NLP tasks.\nAmbiguity\nAmbiguity is an intrinsic characteristic of human conversations and is particularly challenging in Natural Language Understanding scenarios where there might be different forms that are relevant in natural language and in the AI system that we’ve programmed. In AI theory, the process of handling ambiguity is called disambiguation.\nSynonymity\nSynonymity stems from the fact that we can express the same idea with different terms (which are also dependent on the specific context); For example,\n‘big’\nand\n‘large’\nhave a similar meaning when referring to sizes, whereas\n‘large’\ndoesn’t make sense when used as a qualifier to the word\n‘sister’\n.\nCo-reference\nCo-reference is the process of finding all expressions that refer to the same entity in a text.\nCo-Reference resolution is an important step for a lot of higher-level NLP tasks that involve natural language understanding and is often instrumental in improving the performances of neural architectures like RNN and LSTM.\nSyntactic Rules\nKnowledge about the structure and syntax of the language is often helpful. For a more detailed note on the different parsing techniques, please read through my original post:\nNatural Language Processing: Concepts and Workflow\n.\nGeneric NLP Workflow\nThe standard workflow for an NLP problem includes the following steps.\nThe first step is usually text wrangling and pre-processing on the corpus of documents, followed by parsing and basic exploratory data analysis.\nAs the next step, we look at representing text with word embeddings and subsequent feature engineering, followed by choosing the model depending on whether we’re looking at a supervised/unsupervised learning problem.\nAs with any ML workflow, the final stage involves model evaluation and deployment.\nText pre-processing\nContraction Mapping/ Expanding Contractions\nContractions are a shortened version of words or a group of words, quite common in both spoken and written language. In English, they are quite common, such as\nI will\nto\nI’ll\n,\nI have\nto\nI’ve\n,\ndo not\nto\ndon’t\n, etc.\nMapping these contractions to their expanded form helps in text standardization.\nTokenization\nTokenization is the process of separating a piece of text into smaller units called tokens.\nGiven a document, tokens can be sentences, words, subwords, or even characters depending on the application.\nNoise cleaning\nSpecial characters and symbols contribute to extra noise in unstructured text. Using regular expressions to remove them or using tokenizers, which do the pre-processing step of removing punctuation marks and other special characters, is recommended.\nSpell-checking\nDocuments in a corpus are prone to spelling errors; In order to make the text clean for the subsequent processing, it is a good practice to run a spell checker and fix the spelling errors before moving on to the next steps.\nStop words Removal\nStop words are those words which are very common and often less significant. Hence, removing these is a pre-processing step as well.\nThis can be done explicitly by retaining only those words in the document which are not in the list of stop words or by specifying the stop word list as an argument in\nCountVectorizer\nor\nTfidfVectorizer\nmethods when getting Bag-of-Words(BoW)/TF-IDF scores for the corpus of text documents.\nStemming/Lemmatization\nBoth stemming and lemmatization are methods to reduce words to their base form.\nWhile stemming follows certain rules to truncate the words to their base form, often resulting in words that are not lexicographically correct, lemmatization always results in base forms that are lexicographically correct.\nHowever, stemming is a lot faster than lemmatization. Hence, to stem/lemmatize is dependent on whether the application needs quick pre-processing or requires more accurate base forms.\nThis is a very comprehensive introduction to NLP. For a longer read, including an example walkthrough of EDA and text pre-processing on the SMS spam classification dataset, refer to my post\nNLP: Concepts and Workflow\n😀\nReferences and Additional Reading\n[1]\nA Practitioners' Guide to NLP\n[2]\nEDA for Text Data\n[3]\nEDA and Visualization for Text Data\n[4]\nPre-processing Text Data\nCover Image: Photo by\nKimberly Farmer\non\nUnsplash",
  "metadata": {
    "title": "NLP101 — Getting Started with Natural Language Processing - DEV Community",
    "description": "With the huge influx of unstructured text data from a plethora of social media platforms , different... Tagged with nlp, python, machinelearning, beginners.",
    "keywords": "nlp, python, machinelearning, beginners, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/natural-language-processing-101-4226",
    "og:title": "NLP101 — Getting Started with Natural Language Processing",
    "og:description": "With the huge influx of unstructured text data from a plethora of social media platforms , different...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8clifpjlfiaw7zfo0ixg.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "NLP101 — Getting Started with Natural Language Processing",
    "twitter:description": "With the huge influx of unstructured text data from a plethora of social media platforms , different...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8clifpjlfiaw7zfo0ixg.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://towardsai.net/p/nlp/natural-language-processing-concepts-and-workflow-48083d2e3ce7",
      "label": "Natural Language Processing: Concepts and Workflow",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/a-practitioners-guide-to-natural-language-processing-part-i-processing-understanding-text-9f4abfd13e72",
      "label": "A Practitioners' Guide to NLP",
      "type": "a"
    },
    {
      "href": "https://www.analyticsvidhya.com/blog/2020/04/beginners-guide-exploratory-data-analysis-text-data/",
      "label": "EDA for Text Data",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/a-complete-exploratory-data-analysis-and-visualization-for-text-data-29fb1b96fb6a",
      "label": "EDA and Visualization for Text Data",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/preprocessing-text-data-using-python-576206753c28",
      "label": "Pre-processing Text Data",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@kimberlyfarmer?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Kimberly Farmer",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/s/photos/books?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:11Z",
  "source_url": "https://dev.to/balapriya/natural-language-processing-101-4226",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7457,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/bd5215b0744e/pages/source.html",
  "local_html": "attachments/bd5215b0744e/pages/article.html",
  "body_base_path": "attachments/bd5215b0744e/pages/",
  "body_markdown_path": "attachments/bd5215b0744e/pages/article.md"
}