{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "Role of Cross-Validation in Model Validation and Hyperparameter Search",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/role-of-cross-validation-in-model-validation-and-hyperparameter-search-n3",
  "publish_date": "2021-04-04",
  "corpus_stage": "",
  "notes": "",
  "id": "aae4ecf42d64",
  "collected_at": "2026-10-02T14:01:46",
  "title": "Role of Cross-Validation in Model Validation and Hyperparameter Search - DEV Community",
  "body_markdown": "How do we validate a machine learning model?\nWhen evaluating a machine learning model, training and testing on the same dataset is not a great idea; Why?Let us draw a relatable analogy.\nEver since school days, we’ve been giving exams, and how are our exams designed? Well, they’ve been designed so as to test our understanding of the subjects rather than our ability to memorize! The same analogy can be transposed to our machine learning model as well!\nHere’s the answer to the question\n‘Why can we not evaluate a model on the same data that it was trained on?’\nIt is because this process inherently encourages the model to memorize the training data, so it performs extremely well on the training data but generalizes rather poorly and performs badly on the data that it has never seen before, typically overfitting on the training dataset.\nAs a model's performance on data that it has never seen before is a more reliable estimate of its performance, we usually validate the model by checking how it performs on out-of-sample data, that is, on data that it has never seen before. If you remember, it is for this reason, we use the\ntrain_test_split\nmethod, in our very friendly and nifty library, scikit-learn.\nThis\ntrain_test_split\nsplits the available data into two sets: the train and test sets in certain proportion; for example, train on 70% of the available data and test on the remaining 30% of the data. In this way, we can ensure that every record in the dataset can either be in the training set or the test set but not both! In this way, we are making sure that we test the model’s performance on unseen data.\nBut, is this good enough? Or do we take this with a pinch of salt?\nLet us train a simple\nKNeighborsClassifier\nin scikit-learn on the iris dataset. As shown below, we import the necessary modules.\n# Necessary imports\nfrom\nsklearn.datasets\nimport\nload_iris\nfrom\nsklearn.model_selection\nimport\ntrain_test_split\nfrom\nsklearn.neighbors\nimport\nKNeighborsClassifier\nfrom\nsklearn\nimport\nmetrics\nEnter fullscreen mode\nExit fullscreen mode\nLet us load in the iris data and separate out the features(Sepal Length, Sepal Width, Petal Length and Petal Width) and the target variables which are the class labels indicating the iris type. (Setosa, Versicolour, and Virginica)\n# read in the iris data\niris\n=\nload_iris\n()\n# create X (features) and y (response)\nX\n=\niris\n.\ndata\ny\n=\niris\n.\ntarget\nEnter fullscreen mode\nExit fullscreen mode\nLet’s create the train and test sets with\nrandom_state=4\n; Setting the\nrandom_state\nensures reproducibility. In this case, it ensures that the records that went into the train and test sets stay the same every time our code is run.\n# use train/test split with different random_state values\nX_train\n,\nX_test\n,\ny_train\n,\ny_test\n=\ntrain_test_split\n(\nX\n,\ny\n,\nrandom_state\n=\n4\n)\nEnter fullscreen mode\nExit fullscreen mode\nWe now instantiate the\nKNeighborsClassifier\nwith\nn_neighbors=5\nand fit the classifier on the training set and predict on the test set.\n# check classification accuracy of KNN with K=5\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\n5\n)\nknn\n.\nfit\n(\nX_train\n,\ny_train\n)\ny_pred\n=\nknn\n.\npredict\n(\nX_test\n)\nprint\n(\nmetrics\n.\naccuracy_score\n(\ny_test\n,\ny_pred\n))\n# Output\n0.9736842105263158\nEnter fullscreen mode\nExit fullscreen mode\nThe accuracy obtained is\n0.9736842105263158\n. Now, let us change the\nrandom_state\nto a different value, say 20. What do you think the accuracy score would be? It’s now\n0.9473684210526315\n. Setting the\nrandom_state\nto another value, we would get another value for the accuracy score.\nThe evaluation metric thus obtained is therefore susceptible to\nhigh variance\n.\nThe evaluation may depend heavily on which data points end up in the training set and which end up in the test set, and thus the evaluation may be significantly different depending on how the division is made. Clearly, this doesn’t seem like the best way to validate our model’s performance!\nHow do we reach a consensus on how to calculate the accuracy score?\nOne very natural thing to do, would be to create multiple train/test splits, calculate the accuracy for each such split, and compute the average of all the accuracy scores thus obtained. This definitely seems like a\nbetter estimate of the accuracy\n, doesn’t it? This is precisely the\nessence of cross-validation\n, which we shall see in the subsequent section.\nUnderstanding K-fold cross-validation\nSteps in K-fold cross-validation\nSplit the dataset into K equal partitions (or “folds”).\nUse fold 1 for testing and the union of the other folds as the training set.\nCalculate accuracy on the test set.\nRepeat steps 2 and 3 K times, using a different fold for testing each time.\nUse the average accuracy on different test sets as the estimate of out-of-sample accuracy.\nLet us try to visualize this by splitting a dataset of 25 observations into 5 equal folds as shown below. Dataset contains 25 observations (numbered 0 through 24). 5-fold cross-validation runs for 5 iterations.\n# simulate splitting a dataset of 25 observations into 5 folds\nfrom\nsklearn.model_selection\nimport\nKFold\nkf\n=\nKFold\n(\nn_splits\n=\n5\n,\nshuffle\n=\nFalse\n).\nsplit\n(\nrange\n(\n25\n))\n# print the contents of each training and testing set\nprint\n(\n'\n{} {:^61} {}\n'\n.\nformat\n(\n'\nIteration\n'\n,\n'\nTraining set observations\n'\n,\n'\nTesting set observations\n'\n))\nfor\niteration\n,\ndata\nin\nenumerate\n(\nkf\n,\nstart\n=\n1\n):\nprint\n(\n'\n{:^9} {} {:^25}\n'\n.\nformat\n(\niteration\n,\ndata\n[\n0\n],\nstr\n(\ndata\n[\n1\n])))\n# Output\nIteration\nTraining\nset\nobservations\nTesting\nset\nobservations\n1\n[\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n]\n[\n0\n1\n2\n3\n4\n]\n2\n[\n0\n1\n2\n3\n4\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n]\n[\n5\n6\n7\n8\n9\n]\n3\n[\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n]\n[\n10\n11\n12\n13\n14\n]\n4\n[\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n20\n21\n22\n23\n24\n]\n[\n15\n16\n17\n18\n19\n]\n5\n[\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n]\n[\n20\n21\n22\n23\n24\n]\nEnter fullscreen mode\nExit fullscreen mode\nWe observe the following:\nFor each iteration, every observation is either in the training set or the testing set, but not both.\nEvery observation is in the test set exactly once.\nEach fold is used as the test set exactly once and in the training set (K-1) times.\nThe average accuracy thus obtained is a more accurate estimate of out-of-sample accuracy. This process uses data more efficiently as every observation is used for both training and testing. It is recommended to use\nstratified sampling\nfor creating the folds, as this ensures that all class labels are represented in equal proportions in each fold. And, scikit-learn’s\ncross_val_score\ndoes this by default.\nIn practice, we can do even better by doing the following:\n“Hold out” a portion of the data before beginning the model building process.\nFind the best model using cross-validation on the remaining data, and test it using the hold-out set.\nThis gives a more reliable estimate of out-of-sample performance since hold-out set is truly out-of-sample.\nCross-validation for hyperparameter tuning\nFor the KNN classifier on the iris dataset, can we possibly use cross-validation to find the optimal value for\nK\n? That is, to search for the optimal value of\nn_neighbors\n?\nRemember,\nK\nin KNN classifier is the number of neighbors (\nn_neighbors\n) that we take into account for predicting the class label of the test sample. Not to be confused with the K in K-fold cross-validation.\nfrom\nsklearn.model_selection\nimport\ncross_val_score\n# 10-fold cross-validation with K=5 for KNN (the n_neighbors parameter)\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\n5\n)\nscores\n=\ncross_val_score\n(\nknn\n,\nX\n,\ny\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n)\nprint\n(\nscores\n)\n# Output\n[\n1.\n0.93333333\n1.\n1.\n0.86666667\n0.93333333\n0.93333333\n1.\n1.\n1.\n]\n# use average accuracy as an estimate of out-of-sample accuracy\nprint\n(\nscores\n.\nmean\n())\n# Output\n0.9666666666666668\nEnter fullscreen mode\nExit fullscreen mode\nNow, we shall run the K fold cross-validation for the models with different values of\nn_neighbors\n, as shown below.\n# search for an optimal value of K for KNN\nk_range\n=\nlist\n(\nrange\n(\n1\n,\n31\n))\nk_scores\n=\n[]\nfor\nk\nin\nk_range\n:\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\nk\n)\nscores\n=\ncross_val_score\n(\nknn\n,\nX\n,\ny\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n)\nk_scores\n.\nappend\n(\nscores\n.\nmean\n())\nprint\n(\nk_scores\n)\n# Output k_scores\n[\n0.96\n,\n0.9533333333333334\n,\n0.9666666666666666\n,\n0.9666666666666666\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9733333333333334\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9733333333333334\n,\n0.9800000000000001\n,\n0.9733333333333334\n,\n0.9733333333333334\n,\n0.9733333333333334\n,\n0.9733333333333334\n,\n0.9800000000000001\n,\n0.9733333333333334\n,\n0.9800000000000001\n,\n0.9666666666666666\n,\n0.9666666666666666\n,\n0.9733333333333334\n,\n0.96\n,\n0.9666666666666666\n,\n0.96\n,\n0.9666666666666666\n,\n0.9533333333333334\n,\n0.9533333333333334\n,\n0.9533333333333334\n]\nEnter fullscreen mode\nExit fullscreen mode\nThat’s so hard to see; Let us plot the values to get a better idea.\nimport\nmatplotlib.pyplot\nas\nplt\n%\nmatplotlib\ninline\n# plot the value of K for KNN (x-axis) versus the cross-validated accuracy (y-axis)\nplt\n.\nplot\n(\nk_range\n,\nk_scores\n)\nplt\n.\nxlabel\n(\n'\nValue of K for KNN\n'\n)\nplt\n.\nylabel\n(\n'\nCross-Validated Accuracy\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\nWe see that\nn_neighbors\n(K) values from 13 to 20 yield higher accuracy, especially K=13,18 and 20.As a larger value of\nK\nyields a less complex model, we choose K=20 .\nThis process of searching for the optimal values of hyperparameters is called hyperparameter tuning.\nIn this example, we chose the values of\nK\nthat resulted in higher mean accuracy score under 10-fold cross validation.\nThis is how cross-validation can be used to search for the best hyperparameters and this process can be done much more efficiently in scikit-learn.\nReferences\n[1] Here’s the link to\nGoogle Colab notebook\nfor the example discussed above.\n[2]\nIntroduction to Machine Learning in Python with scikit-learn\nby DataSchool.\n[3] Scikit-learn Documentation:\nhttp://scikitlearn.org/stable/modules/cross_validation.html\nCover Image: Photo by\nKatie Harp\non\nUnsplash",
  "metadata": {
    "title": "Role of Cross-Validation in Model Validation and Hyperparameter Search - DEV Community",
    "description": "How do we validate a machine learning model?   When evaluating a machine learning model,... Tagged with python, machinelearning, tutorial, beginners.",
    "keywords": "python, machinelearning, tutorial, beginners, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/role-of-cross-validation-in-model-validation-and-hyperparameter-search-n3",
    "og:title": "Role of Cross-Validation in Model Validation and Hyperparameter Search",
    "og:description": "How do we validate a machine learning model?   When evaluating a machine learning model,...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7rzkcxl29521wz06hyqd.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Role of Cross-Validation in Model Validation and Hyperparameter Search",
    "twitter:description": "How do we validate a machine learning model?   When evaluating a machine learning model,...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F7rzkcxl29521wz06hyqd.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/aae4ecf42d64/code/body-000.txt",
      "chars": 191,
      "language": "text"
    },
    {
      "file": "attachments/aae4ecf42d64/code/body-001.txt",
      "chars": 113,
      "language": "text"
    },
    {
      "file": "attachments/aae4ecf42d64/code/body-002.txt",
      "chars": 132,
      "language": "text"
    },
    {
      "file": "attachments/aae4ecf42d64/code/body-003.txt",
      "chars": 220,
      "language": "text"
    },
    {
      "file": "attachments/aae4ecf42d64/code/body-004.txt",
      "chars": 1024,
      "language": "text"
    },
    {
      "file": "attachments/aae4ecf42d64/code/body-005.txt",
      "chars": 482,
      "language": "text"
    },
    {
      "file": "attachments/aae4ecf42d64/code/body-006.txt",
      "chars": 855,
      "language": "text"
    },
    {
      "file": "attachments/aae4ecf42d64/code/body-007.txt",
      "chars": 236,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgxup7pe297uojl1u9h5g.png",
      "bytes": 16249,
      "reused": true,
      "local_path": "attachments/aae4ecf42d64/images/002.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgxup7pe297uojl1u9h5g.png",
      "alt": "Alt Text",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgxup7pe297uojl1u9h5g.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1RI5q9n3YFY6uqE5XZRg4y5nfebDtBq9t?usp=sharing",
      "label": "Google Colab notebook",
      "type": "a"
    },
    {
      "href": "https://www.dataschool.io/machine-learning-with-scikit-learn/",
      "label": "Introduction to Machine Learning in Python with scikit-learn",
      "type": "a"
    },
    {
      "href": "http://scikitlearn.org/stable/modules/cross_validation.html",
      "label": "http://scikitlearn.org/stable/modules/cross_validation.html",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@kharp?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Katie Harp",
      "type": "a"
    },
    {
      "href": "https://dev.to/@tolga__?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:12Z",
  "source_url": "https://dev.to/balapriya/role-of-cross-validation-in-model-validation-and-hyperparameter-search-n3",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 10156,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/aae4ecf42d64/pages/source.html",
  "local_html": "attachments/aae4ecf42d64/pages/article.html",
  "body_base_path": "attachments/aae4ecf42d64/pages/",
  "body_markdown_path": "attachments/aae4ecf42d64/pages/article.md"
}