{
  "source_file": "0930",
  "source_platform": "Hashnode",
  "author": "Muhammad Fahad Bashir",
  "title_listed": "4. Before the Model: Essential Steps of Data Preprocessing in ML",
  "keywords": "",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://ml-diaries-by-fahd.hashnode.dev/4-before-the-model-essential-steps-of-data-preprocessing-in-ml",
  "publish_date": "2025-09-20",
  "corpus_stage": "",
  "notes": "",
  "id": "58d2562888d5",
  "collected_at": "2026-10-02T14:10:55",
  "title": "4. Before the Model: Essential Steps of Data Preprocessing in ML",
  "body_markdown": "4. Before the Model: Essential Steps of Data Preprocessing in ML\nUpdated\nSeptember 20, 2025\n•\n3\nmin read\n•\nView as Markdown\nM\nMuhammad Fahad Bashir\nMachine Learning, Artificial Intelligence\nWhen working on machine learning projects, we often get excited about building and training models. But here’s the truth:\nyour model is only as good as your data.\nIf the data is noisy, inconsistent, or incomplete, no algorithm—no matter how powerful—will give you reliable results. This is where data preprocessing comes in.\nIn this article, we’ll walk through the essential steps of preparing your data before model training.\nWhy Data Preprocessing Matters\nThe quality of your dataset directly affects:\nModel performance\n(better accuracy and predictions)\nTraining efficiency\n(less complexity for the algorithm)\nInterpretability\n(cleaner results that make sense)\nThat’s why preprocessing is often said to take 80% of a data scientist’s time.\nCommonly used libraries for preprocessing:\nPandas\n– for data cleaning and manipulation\nNumPy\n– for numerical computations\nKey Steps in Data Preprocessing\nHandling Missing Data\nReal-world data is often incomplete. For example, in a dataset of student records, some students might have missing grades. We can’t just feed missing values into a model.\nWays to handle missing data:\nReplace missing data with mean, median, or mode\nUse advanced imputation methods\nDrop rows or columns if too many values are missing\n# Example: Fill missing values with the column mean\ndf[\n'Age'\n].fillna(df[\n'Age'\n].mean(), inplace=\nTrue\n)\n# Or drop rows with missing values\ndf.dropna(inplace=\nTrue\n)\nDetecting and Treating Outliers\nOutliers are unusual values that don’t fit the general pattern. For example:\nAge = 500 years in a human dataset.\nWhy remove them?\nThey can\nskew result\ns and make models less accurate.\nTechniques include:\nUsing boxplots or z-scores to detect outliers\nRemoving or adjusting them\nData Cleaning\nData may have duplicates, typos, or irrelevant values. For instance, two identical rows in a sales dataset can bias results.\nSteps:\nRemove duplicates\nStandardize formats (e.g., date formats, units like “kg” vs “kilograms”)\n# Remove duplicate rows\ndf.drop_duplicates(inplace=\nTrue\n)\nFeature Engineering\nNot all features add value. Sometimes, combining or transforming features improves model performance.\nExamples:\nCreating a new feature like BMI = weight / height²\nDropping irrelevant features like “student ID”\nSelecting only features that affect predictions\n# Example: Create BMI from height & weight\ndf[\n'BMI'\n] = df[\n'Weight'\n] / (df[\n'Height'\n]/\n100\n)**\n2\n# Drop irrelevant columns\ndf.drop(columns=[\n'Student_ID'\n], inplace=\nTrue\n)\nEncoding Categorical Data\nMachine learning algorithms work with numbers, not text. So categorical data (like “Male/Female” or “Yes/No”) must be converted.\nCommon techniques:\nLabel Encoding\n: Assigns numbers to categories (e.g., Male=0, Female=1)\nOne-Hot Encoding\n: Creates separate binary columns for each category\nOrdinal Encoding\n: Useful when categories have an order (e.g., Small=1, Medium=2, Large=3)\nFeature Scaling\nFeatures like salary (in thousands) and age (in years) exist on different scales. Models may give more importance to larger values unless we scale them.\nPopular techniques:\nMin-Max Scaling\n– converts values into a fixed range (e.g., 0–1)\nStandardization\n– rescales data to have mean=0 and standard deviation=1\nSplitting Data into Train and Test Sets\nOnce preprocessing is complete, we split the dataset:\nTraining set (70–80%)\n– used to train the model\nTesting set (20–30%)\n– used to evaluate performance\nThis ensures the model doesn’t just “memorize” data but generalizes well to unseen data.\nfrom\nsklearn.model_selection\nimport\ntrain_test_split\n\nX = df.drop(\n'Target'\n, axis=\n1\n)\n# features\ny = df[\n'Target'\n]\n# target variable\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, test_size=\n0.3\n, random_state=\n42\n)\nTarget Variable vs Input Features\nTarget Variable (y):\nThe outcome we want to predict\nExample\n: Whether a patient has cancer (Yes/No)\nInput Features (X)\n: Independent variables used for prediction\nExample\n: Age, blood pressure, cholesterol level\nFinal Thoughts\nData preprocessing may seem less exciting than building models, but it’s the most critical step in any ML pipeline. If you invest time in preparing high-quality data, your models will reward you with accuracy, stability, and reliability.\n#\nmachine-learnin\n#\ndata-analysis\n#\neda\n#\npython\n#\nbeginners\n#\ntutorial\n8\nviews",
  "metadata": {
    "title": "4. Before the Model: Essential Steps of Data Preprocessing in ML",
    "description": "Preprocess data for reliable and accurate machine learning predictions by ensuring cleanliness and consistency",
    "keywords": null,
    "author": null,
    "og:title": "4. Before the Model: Essential Steps of Data Preprocessing in ML",
    "og:description": "Preprocess data for reliable and accurate machine learning predictions by ensuring cleanliness and consistency",
    "og:url": "https://ml-diaries-by-fahd.hashnode.dev/4-before-the-model-essential-steps-of-data-preprocessing-in-ml",
    "og:site_name": "ML Diaries by Fahd",
    "og:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1758353682386/3bd2c245-6e4d-4bdd-b065-ad86de1cbda6.png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "4. Before the Model: Essential Steps of Data Preprocessing in ML",
    "twitter:description": "Preprocess data for reliable and accurate machine learning predictions by ensuring cleanliness and consistency",
    "twitter:image": "https://cdn.hashnode.com/res/hashnode/image/upload/v1758353682386/3bd2c245-6e4d-4bdd-b065-ad86de1cbda6.png",
    "article:published_time": "2025-09-20T07:32:27.437Z",
    "article:modified_time": "2025-09-20T07:35:17.662Z",
    "article:author": "https://ml-diaries-by-fahd.hashnode.dev/@mfahadbashir"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/58d2562888d5/code/body-000.txt",
      "chars": 161,
      "language": "text"
    },
    {
      "file": "attachments/58d2562888d5/code/body-001.txt",
      "chars": 57,
      "language": "text"
    },
    {
      "file": "attachments/58d2562888d5/code/body-002.txt",
      "chars": 173,
      "language": "text"
    },
    {
      "file": "attachments/58d2562888d5/code/body-003.txt",
      "chars": 242,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://cdn.hashnode.com/res/hashnode/image/upload/v1758353682386/3bd2c245-6e4d-4bdd-b065-ad86de1cbda6.png",
      "bytes": 1224757,
      "reused": true,
      "local_path": "attachments/58d2562888d5/images/002.png",
      "src": "https://cdn.hashnode.com/res/hashnode/image/upload/v1758353682386/3bd2c245-6e4d-4bdd-b065-ad86de1cbda6.png",
      "alt": "4. Before the Model: Essential Steps of Data Preprocessing in ML",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media.geeksforgeeks.org/wp-content/uploads/20250829154818842988/data_preprocessing.webp",
      "bytes": 17650,
      "reused": true,
      "local_path": "attachments/58d2562888d5/images/004.webp",
      "src": "https://media.geeksforgeeks.org/wp-content/uploads/20250829154818842988/data_preprocessing.webp",
      "alt": "Data Preprocessing in Python - GeeksforGeeks",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://builtin.com/sites/www.builtin.com/files/styles/ckeditor_optimize/public/inline-images/4_train-test-split.jpg",
      "bytes": 32763,
      "reused": true,
      "local_path": "attachments/58d2562888d5/images/005.jpg",
      "src": "https://builtin.com/sites/www.builtin.com/files/styles/ckeditor_optimize/public/inline-images/4_train-test-split.jpg",
      "alt": "Train Test Split: What it Means and How to Use It | Built In",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://ml-diaries-by-fahd.hashnode.dev/4-before-the-model-essential-steps-of-data-preprocessing-in-ml.md",
      "label": "View as Markdown",
      "type": "a"
    },
    {
      "href": "https://hashnode.com/@mfahadbashir",
      "label": "M",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/muhammad-fahad-bashir-09568216a/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/machine-learnin",
      "label": "# machine-learnin",
      "type": "a"
    },
    {
      "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/data-analysis",
      "label": "# data-analysis",
      "type": "a"
    },
    {
      "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/eda",
      "label": "# eda",
      "type": "a"
    },
    {
      "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/python",
      "label": "# python",
      "type": "a"
    },
    {
      "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/beginners",
      "label": "# beginners",
      "type": "a"
    },
    {
      "href": "https://ml-diaries-by-fahd.hashnode.dev/tag/tutorial",
      "label": "# tutorial",
      "type": "a"
    }
  ],
  "retry_at": "2026-10-03T08:07:45",
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:43Z",
  "source_url": "https://ml-diaries-by-fahd.hashnode.dev/4-before-the-model-essential-steps-of-data-preprocessing-in-ml",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 4497,
  "expected_body_images": 3,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/58d2562888d5/pages/source.html",
  "local_html": "attachments/58d2562888d5/pages/article.html",
  "body_base_path": "attachments/58d2562888d5/pages/",
  "body_markdown_path": "attachments/58d2562888d5/pages/article.md"
}