{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "amananandrai",
  "title_listed": "10 Important NLP Tasks Commonly Used in Language Modeling",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/amananandrai/10-important-nlp-tasks-commonly-used-in-language-modeling-3ppa",
  "publish_date": "2021-03-23",
  "corpus_stage": "",
  "notes": "",
  "id": "0dc981c59f96",
  "collected_at": "2026-10-02T14:02:22",
  "title": "10 Important NLP Tasks Commonly Used in Language Modeling - DEV Community",
  "body_markdown": "Language Models are machine learning models that work on text data to perform different tasks related to Natural Language Processing. NLP has basically two major categories of tasks\nNatural Language Understanding\nand\nNatural Language Generation\n. There are many different tasks performed by language models that include: Sentiment Analysis, Question-Answering, Query Resolution, Text Summarization, etc. There are many intermediate tasks that are performed to make the language models better. Some of these are given below-\nSentence segmentation\nThe whole 'corpus' which refers to the entire text collection is broken down into separate sentences. This is the first step in understanding languages first they are broken into simple sentences.\nTokenization\nThe next step after sentence segmentation is tokenization or more correctly word tokenization. The sentences are broken down into words, in some of the tasks where there is the importance of punctuation marks they are also treated as tokens along with words.\nStemming\nStemming refers to the process of reducing the words to their root stem, it is done by chopping off the end of the words.\neg- oppressor, oppression, oppressed, oppressive will all we changed to oppress which is the root stem by chopping off 'or', 'ion',' ed', 'ive' respectively from each word.\nSome of the famous stemming algorithms are- Porter Stemmer, Lancaster Stemmer, and Snowball Stemmer. These are implemented in the\nnltk\nlibrary and the packages can be imported as follow-\nfrom nltk.stem.porter import PorterStemmer\n,\nfrom nltk.stem.lancaster import LancasterStemmer\n,\nfrom nltk.stem import SnowballStemmer\n.\nLemmatizing\nLemmatization means to convert the word to its root word known as lemma it takes into account the meaning of the word and just does not simply chop off the last section of words.\neg- the root word for 'is', 'am', 'are' is 'be',another example is\nthe words 'creation', 'creating', 'creative' will be changed to 'create'. If stemming was done then they will become 'creati' instead of 'create'.\nPOS tags\nPOS means Part Of Speech and in this step, all the tokens are assigned or tagged with a part of speech. POS tagging has basically two methods -\nRule Based POS Tagging\nand\nStochastic Based POS Tagging\n. POS tagging is used because it helps in building lemmatizers, it helps in building parse trees which are used for Named Entity Recognition, and also resolving word disambiguation.\nIdentifying Stopwords\nStopwords are common words found in languages that do not give a lot of meaning to the sentence like 'and', 'the', 'is', 'am', etc. These words must be identified and based on task removed from the corpus because they are like noise in the dataset.\nName Entity Recognition\nNamed-entity recognition (NER) is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.\nSome common NER tools are-\nStanford Named Entity Recognizer (SNER)\n,\nSpaCy\n,\nNatural Language Toolkit (NLTK)\n.\nText Classification\nText Classification is one of the most important steps in Sentiment Analysis after all the steps like tokenization, stemming and lemmatization are performed on the corpus they are passed to any machine learning algorithm to classify it.\nChunking\nIt works on top of POS tagging. It uses POS-tags as input and provides chunks as output. In short, Chunking means grouping of words/tokens into chunks. The chunks are a group of words or phrases which can be clubbed together to form meaningful parts of the sentence like noun group/phrase, verb group/phrase, etc.\nChunking can break sentences into phrases that are more useful than individual words and yield meaningful results. Chunking is very important when you want to extract information from text such as locations, person names (NER). NLTK can be used for chunking.\nCoreference Resolution\nIn linguistics, coreference occurs when two or more expressions in a text refer to the same person or thing; they have the same referent, e.g. Bill said he would come; the proper noun Bill and the pronoun he refers to the same person, namely to Bill. Coreference resolution is the task of finding all expressions that refer to the same entity in a text. It is an important step for a lot of higher-level NLP tasks that involve natural language understanding such as document summarization, question answering, and information extraction.",
  "metadata": {
    "title": "10 Important NLP Tasks Commonly Used in Language Modeling - DEV Community",
    "description": "Language Models are machine learning models that work on text data to perform different tasks related... Tagged with machinelearning, nlp.",
    "keywords": "machinelearning, nlp, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/amananandrai/10-important-nlp-tasks-commonly-used-in-language-modeling-3ppa",
    "og:title": "10 Important NLP Tasks Commonly Used in Language Modeling",
    "og:description": "Language Models are machine learning models that work on text data to perform different tasks related...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ffyp7reqehjwpab74c9pv.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "10 Important NLP Tasks Commonly Used in Language Modeling",
    "twitter:description": "Language Models are machine learning models that work on text data to perform different tasks related...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ffyp7reqehjwpab74c9pv.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:18Z",
  "source_url": "https://dev.to/amananandrai/10-important-nlp-tasks-commonly-used-in-language-modeling-3ppa",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4549,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/0dc981c59f96/pages/source.html",
  "local_html": "attachments/0dc981c59f96/pages/article.html",
  "body_base_path": "attachments/0dc981c59f96/pages/",
  "body_markdown_path": "attachments/0dc981c59f96/pages/article.md"
}