{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "How to Handle Missing Data Better [A scikit-learn Tutorial]",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/having-trouble-with-missing-data-here-s-how-you-can-handle-them-better-42d6",
  "publish_date": "2021-03-07",
  "corpus_stage": "",
  "notes": "",
  "id": "1988c8554fbd",
  "collected_at": "2026-10-02T14:01:49",
  "title": "How to Handle Missing Data Better [A scikit-learn Tutorial] - DEV Community",
  "body_markdown": "Why is Data Cleaning Important?\nData from the real world is often messy and cleaning up the data is an integral step in any machine learning project. To ensure optimal model performance, data cleaning is important as machine learning algorithms are sensitive to the quality of data that we feed in.\nDealing with missing data, experimenting with the\noptimal imputation strategies\nand ensuring that the data is ready for use in the rest of the pipeline is therefore crucial.\nIn this blog post, we shall see useful features of\nscikit-learn\nthat help us handle missing data all the more gracefully! In general, how do we handle missing values in input data? The following are the usual approaches.\nBy dropping columns containing NaNs.\nBy dropping rows containing NaNs.\nBy imputing the missing values suitably.\nWouldn’t it be cool if we could do the following instead?\nEncode ‘missingness’ as a feature.\nUse\nHistGradientBoostingClassifier\nthat automatically imputes missing values.\nEncoding ‘Missingness’ as a Feature\nWhen imputing missing values, if we would like to preserve information about which values were missing and would like to use that as a feature, then we can do it by setting the\nadd_indicator\nattribute in scikit-learn’s\nSimpleImputer\nto\nTrue\n. Here’s an example. Let’s import numpy and pandas using their usual aliases\nnp\nand\npd\n.\n# Necessary imports\nimport\nnumpy\nas\nnp\nimport\npandas\nas\npd\nEnter fullscreen mode\nExit fullscreen mode\nLet’s create a pandas DataFrame X with one missing value.\nX\n=\npd\n.\nDataFrame\n({\n'\nAge\n'\n:[\n20\n,\n30\n,\n10\n,\nnp\n.\nnan\n,\n10\n]})\nEnter fullscreen mode\nExit fullscreen mode\nNow, we shall import the\nSimpleImputer\nfrom\nscikit-learn\nfrom\nsklearn.impute\nimport\nSimpleImputer\nEnter fullscreen mode\nExit fullscreen mode\nWe shall now instantiate a\nSimpleImputer\nthat by default does\nmean imputation\n, by replacing all missing values with the average of the other values present. The missing value is calculated as (20+30+10+10)/4=17.5. Let's verify the output.\n# Mean Imputation\nimputer\n=\nSimpleImputer\n()\nimputer\n.\nfit_transform\n(\nX\n)\n# After Imputation\narray\n([[\n20.\n],\n[\n30.\n],\n[\n10.\n],\n[\n17.5\n],\n[\n10.\n]])\nEnter fullscreen mode\nExit fullscreen mode\nIn order to encode the missingness of values as a feature, we can set the\nadd_indicator\nargument to\nTrue\nand observe the output.\n# impute the mean and add an indicator matrix (new in scikit-learn 0.21)\nimputer\n=\nSimpleImputer\n(\nadd_indicator\n=\nTrue\n)\nimputer\n.\nfit_transform\n(\nX\n)\n# After adding missingness indicator\narray\n([[\n20.\n,\n0.\n],\n[\n30.\n,\n0.\n],\n[\n10.\n,\n0.\n],\n[\n17.5\n,\n1.\n],\n[\n10.\n,\n0.\n]])\nEnter fullscreen mode\nExit fullscreen mode\nIn the output, we observe that the indicator value of 1 is inserted at index 3 where the original data was missing. This feature is new in\nscikit-learn\nversion 0.21 and above. In the next section, we shall see how we can use the\nHistGradientBoosting Classifier\nthat natively handles missing values.\nUsing HistGradientBoosting Classifier\nTo use this new feature in\nscikit-learn\nversion 0.22 and above, let’s download the very popular Titanic-Machine learning from Disaster dataset from kaggle.\nimport\npandas\nas\npd\ntrain\n=\npd\n.\nread_csv\n(\n'\nhttp://bit.ly/kaggletrain\n'\n)\ntest\n=\npd\n.\nread_csv\n(\n'\nhttp://bit.ly/kaggletest\n'\n,\nnrows\n=\n175\n)\nEnter fullscreen mode\nExit fullscreen mode\nNow that we’ve imported the dataset, let’s go ahead and create the datasets for training and testing.\ntrain\n=\ntrain\n[[\n'\nSurvived\n'\n,\n'\nAge\n'\n,\n'\nFare\n'\n,\n'\nPclass\n'\n]]\ntest\n=\ntest\n[[\n'\nAge\n'\n,\n'\nFare\n'\n,\n'\nPclass\n'\n]]\nEnter fullscreen mode\nExit fullscreen mode\nTo better understand the missing values, let’s compute the number of missing values in each column of the training and test sets.\n# count the number of NaNs in each column\nprint\n(\ntrain\n.\nisna\n().\nsum\n())\nSurvived\n0\nAge\n177\nFare\n0\nPclass\n0\ndtype\n:\nint64\nprint\n(\ntest\n.\nisna\n().\nsum\n())\nAge\n36\nFare\n1\nPclass\n0\ndtype\n:\nint64\nEnter fullscreen mode\nExit fullscreen mode\nWe see that both train and test subsets contain missing values. Let the output label for the classifier be\nSurvived\nindicated by\n1\nif the passenger survived and\n0\nif the passenger did not.\nlabel\n=\ntrain\n.\npop\n(\n'\nSurvived\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\nLet’s import\nHistGradientBoostingClassifier\nfrom scikit-learn.\nfrom\nsklearn.experimental\nimport\nenable_hist_gradient_boosting\nfrom\nsklearn.ensemble\nimport\nHistGradientBoostingClassifier\nEnter fullscreen mode\nExit fullscreen mode\nAs always, let us instantiate the classifier, fit on the training set train and predict on the test set test . Note that we did not impute the missing values; Ideally, when there are missing values NaN, we do get errors. Let us check what happens now.\nclf\n=\nHistGradientBoostingClassifier\n()\n# no errors, despite NaNs in train and test sets!\nclf\n.\nfit\n(\ntrain\n,\nlabel\n)\nclf\n.\npredict\n(\ntest\n)\n# Output\narray\n([\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n1\n,\n0\n,\n1\n,\n1\n,\n0\n,\n1\n,\n0\n,\n1\n,\n1\n,\n1\n,\n0\n,\n1\n,\n1\n,\n1\n,\n0\n,\n1\n,\n1\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n1\n,\n1\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n1\n,\n0\n,\n1\n,\n0\n,\n1\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n1\n,\n0\n,\n1\n,\n0\n,\n1\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n1\n,\n0\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n1\n,\n1\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n1\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n1\n,\n0\n,\n0\n,\n1\n,\n0\n,\n1\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n,\n0\n])\nEnter fullscreen mode\nExit fullscreen mode\nSurprisingly, there are no errors and we get predictions for all records in the test set even though there were missing values. Isn’t this cool? Be sure to try out these features in your next project.\nHappy Learning!\nReferences\n[1]\nUseful scikit-learn tips by Kevin Markham from DataSchool\nCover Image:\nPhoto by\nAndrew Neel\non\nUnsplash",
  "metadata": {
    "title": "How to Handle Missing Data Better [A scikit-learn Tutorial] - DEV Community",
    "description": "Why is Data Cleaning Important?   Data from the real world is often messy and cleaning up... Tagged with python, machinelearning, datascience, beginners.",
    "keywords": "python, machinelearning, datascience, beginners, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/having-trouble-with-missing-data-here-s-how-you-can-handle-them-better-42d6",
    "og:title": "How to Handle Missing Data Better [A scikit-learn Tutorial]",
    "og:description": "Why is Data Cleaning Important?   Data from the real world is often messy and cleaning up...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkolzhkjgtcar4w20em7x.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "How to Handle Missing Data Better [A scikit-learn Tutorial]",
    "twitter:description": "Why is Data Cleaning Important?   Data from the real world is often messy and cleaning up...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkolzhkjgtcar4w20em7x.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/1988c8554fbd/code/body-000.txt",
      "chars": 59,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-001.txt",
      "chars": 51,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-002.txt",
      "chars": 41,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-003.txt",
      "chars": 165,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-004.txt",
      "chars": 286,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-005.txt",
      "chars": 127,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-006.txt",
      "chars": 92,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-007.txt",
      "chars": 180,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-008.txt",
      "chars": 30,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-009.txt",
      "chars": 123,
      "language": "text"
    },
    {
      "file": "attachments/1988c8554fbd/code/body-010.txt",
      "chars": 728,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/justmarkham/scikit-learn-tips",
      "label": "Useful scikit-learn tips by Kevin Markham from DataSchool",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@andrewtneel?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Andrew Neel",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/s/photos/data?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:14Z",
  "source_url": "https://dev.to/balapriya/having-trouble-with-missing-data-here-s-how-you-can-handle-them-better-42d6",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5922,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/1988c8554fbd/pages/source.html",
  "local_html": "attachments/1988c8554fbd/pages/article.html",
  "body_base_path": "attachments/1988c8554fbd/pages/",
  "body_markdown_path": "attachments/1988c8554fbd/pages/article.md"
}