{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "Cross-Validation and Hyperparameter Search in Scikit-Learn - A Complete Guide",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/cross-validation-and-hyperparameter-search-in-scikit-learn-a-complete-guide-5ed8",
  "publish_date": "2021-08-24",
  "corpus_stage": "",
  "notes": "",
  "id": "6c5a439f68cd",
  "collected_at": "2026-10-02T14:01:28",
  "title": "Cross-Validation and Hyperparameter Search in Scikit-Learn - A Complete Guide - DEV Community",
  "body_markdown": "This post is inspired by Kevin Markham's course\nIntroduction to Machine Learning with scikit-learn\non Data School. If you're a beginner looking to get started with Machine Learning using scikit-learn, I would highly recommend this course to gain all the required foundational skills.\nIn just a few hours, you'd be able to understand and build basic regression, and classification models with the optimal hyperparameters.\n📌If this sounds interesting, please be sure to check the course at\nthis\nlink.\nThat said, we shall cover the following in this post.\nHow to Use Cross-Validation to Evaluate a Machine Learning Model\nHow to Use Cross-Validation to Search for the Best Model Hyperparameters\nHow to Use Grid Search for Hyperparameter Search\nHow to Search for Hyperparameters More Efficiently Using Randomized Search\nHow to Use Cross-Validation to Evaluate a Machine Learning Model\nHow do we validate a machine learning model?\nWhen evaluating a machine learning model, training and testing on the same dataset is not a great idea. Why? Let us draw a relatable analogy.\nEver since school days, we’ve been giving exams, and how are our exams designed?\nWell, they’ve been designed so as to test our understanding of the concepts rather than our ability to memorize!\nThe same analogy can be transposed to our machine learning model as well!\nHere’s the answer to the question\n‘Why can we not evaluate a model on the same data that it was trained on?’\nIt is because this process inherently encourages the model to\nmemorize\nthe training data. So, the model performs extremely well on the training data.\nHowever, it\ngeneralizes rather poorly\nto data that it has never seen before.\nWhat does\ntrain_test_split\ndo?\nAs a model's performance on data that it has never seen before is a more reliable estimate of its performance, we usually validate the model by checking how it performs on out-of-sample data, that is, on data that it has never seen before.\nIf you remember, it is for this reason, we use the\ntrain_test_split\nmethod, in our very friendly and nifty library, scikit-learn.\nThis\ntrain_test_split\nsplits the available data into two sets: the train and test sets in certain proportions.\nFor example, train on 70% of the available data and test on the remaining 30% of the data. This way, we can ensure that every record in the dataset can either be in the training set or the test set but not both!\nAnd in doing so, we are making sure that we test the model’s performance on unseen data.\nBut, is this good enough? Or do we take this with a pinch of salt?\nLet us train a simple\nKNeighborsClassifier\nin scikit-learn on the iris dataset.\n▶\nNecessary imports\nAs shown below, we import the necessary modules.\nfrom\nsklearn.datasets\nimport\nload_iris\nfrom\nsklearn.model_selection\nimport\ntrain_test_split\nfrom\nsklearn.neighbors\nimport\nKNeighborsClassifier\nfrom\nsklearn\nimport\nmetrics\nEnter fullscreen mode\nExit fullscreen mode\n▶\nLoad the data\nLet us load the iris data and separate out the\nfeatures\n(Sepal Length, Sepal Width, Petal Length and Petal Width) and the\ntarget\nvariables which are the class labels indicating the iris type. (Setosa, Versicolour, and Virginica)\n# read in the iris data\niris\n=\nload_iris\n()\n# create X (features) and y (response)\nX\n=\niris\n.\ndata\ny\n=\niris\n.\ntarget\nEnter fullscreen mode\nExit fullscreen mode\n▶\nCreate train_test_split\nLet’s create the train and test sets with\nrandom_state= 4\n. Setting the\nrandom_state\nensures reproducibility.\nIn this case, it ensures that the records that go into the train and test sets stay the same every time our code is run.\n# use train/test split with different random_state values\nX_train\n,\nX_test\n,\ny_train\n,\ny_test\n=\ntrain_test_split\n(\nX\n,\ny\n,\nrandom_state\n=\n4\n)\nEnter fullscreen mode\nExit fullscreen mode\n▶\nCheck Classification Accuracy\nWe now instantiate the\nKNeighborsClassifier\nwith\nn_neighbors=9\nand fit the classifier on the training set and predict on the test set.\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\n9\n)\nknn\n.\nfit\n(\nX_train\n,\ny_train\n)\ny_pred\n=\nknn\n.\npredict\n(\nX_test\n)\nprint\n(\nmetrics\n.\naccuracy_score\n(\ny_test\n,\ny_pred\n))\n# Output\n0.9736842105263158\nEnter fullscreen mode\nExit fullscreen mode\nNow, let us change the\nrandom_state\nto a different value. What do you think the accuracy score would be?\nWell, please feel free to insert your favorite number in the\nrandom_state\nand check for yourselves. It'd be a different accuracy score this time.\nSetting the\nrandom_state\nto another value, we would get another value for the accuracy score.\nThe evaluation metric thus obtained is therefore susceptible to\nhigh variance\n.\nThe accuracy score may depend heavily on which data points end up in the training set and which end up in the test set.\nThus the accuracy score may be significantly different depending on how the division is made. Clearly, this doesn’t seem like the best way to validate our model’s performance!\nHow do we reach a consensus on how to calculate the accuracy score?\nOne very natural thing to do would be to create\nmultiple train/test splits\n, calculate the accuracy for each such split, and compute the average of all the accuracy scores thus obtained. This definitely seems like a better estimate of the accuracy, doesn’t it?\nThis is precisely the essence of cross-validation, which we shall see in the subsequent section.\nUnderstanding K-fold cross-validation\nSteps in K-fold cross-validation\nSplit the dataset into K equal partitions (or “folds”).\nUse fold 1 for testing and the union of the other folds as the training set.\nCalculate accuracy on the test set.\nRepeat steps 2 and 3 K times, using a different fold for testing each time.\nUse the average accuracy on different test sets as the estimate of out-of-sample accuracy.\nLet us try to visualize this by splitting a dataset of 25 observations into 5 equal folds as shown below.\nThe dataset contains 20 observations (numbered 0 through 24).\n5-fold cross-validation runs for 5 iterations.\nLet's see how the folds are created\n# simulate splitting a dataset of 25 observations into 5 folds\nfrom\nsklearn.model_selection\nimport\nKFold\nkf\n=\nKFold\n(\nn_splits\n=\n5\n,\nshuffle\n=\nFalse\n).\nsplit\n(\nrange\n(\n25\n))\n# print the contents of each training and testing set\nprint\n(\n'\n{} {:^61} {}\n'\n.\nformat\n(\n'\nIteration\n'\n,\n'\nTraining set observations\n'\n,\n'\nTesting set observations\n'\n))\nfor\niteration\n,\ndata\nin\nenumerate\n(\nkf\n,\nstart\n=\n1\n):\nprint\n(\n'\n{:^9} {} {:^25}\n'\n.\nformat\n(\niteration\n,\ndata\n[\n0\n],\nstr\n(\ndata\n[\n1\n])))\n# Output\nIteration\nTraining\nset\nobservations\nTesting\nset\nobservations\n1\n[\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n]\n[\n0\n1\n2\n3\n4\n]\n2\n[\n0\n1\n2\n3\n4\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n]\n[\n5\n6\n7\n8\n9\n]\n3\n[\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n]\n[\n10\n11\n12\n13\n14\n]\n4\n[\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n20\n21\n22\n23\n24\n]\n[\n15\n16\n17\n18\n19\n]\n5\n[\n0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n]\n[\n20\n21\n22\n23\n24\n]\nEnter fullscreen mode\nExit fullscreen mode\nWe observe the following:\nFor each iteration, every observation is either in the training set or the testing set, but not both.\nEvery observation is in the test set exactly once.\nEach fold is used as the test set exactly once and in the training set (K-1) times.\nThe average accuracy thus obtained is a more accurate estimate of out-of-sample accuracy.\nThis process uses data more efficiently as every observation is used for both training and testing.\nIt is recommended to use\nstratified sampling\nfor creating the folds, as this ensures that\nall class labels\nare represented in\nequal proportions\nin\neach fold\n. And, scikit-learn’s\ncross_val_score\ndoes this by default.\nIn practice, we can even do the following:\n“Hold out” a portion of the data before beginning the model building process.\nFind the best model using cross-validation on the remaining data, and test it using the hold-out set.\nThis gives a more reliable estimate of out-of-sample performance since the hold-out set is truly out-of-sample.\nHow to Use Cross-Validation to Search for the Best Model Hyperparameters\nCross-validation for hyperparameter tuning\nFor the KNN classifier on the iris dataset, can we possibly use cross-validation to find the optimal value for\nK\n? That is, to search for the optimal value of\nn_neighbors\n?\nRemember,\nK\nin KNN classifier is the number of neighbors (\nn_neighbors\n) that we take into account for predicting the class label of the test sample. Not to be confused with the\nK\nin K-fold cross-validation.\nfrom\nsklearn.model_selection\nimport\ncross_val_score\n# 10-fold cross-validation with K=5 for KNN (the n_neighbors parameter)\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\n5\n)\nscores\n=\ncross_val_score\n(\nknn\n,\nX\n,\ny\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n)\nprint\n(\nscores\n)\n# Output\n[\n1.\n0.93333333\n1.\n1.\n0.86666667\n0.93333333\n0.93333333\n1.\n1.\n1.\n]\n# use average accuracy as an estimate of out-of-sample accuracy\nprint\n(\nscores\n.\nmean\n())\n# Output\n0.9666666666666668\nEnter fullscreen mode\nExit fullscreen mode\nNow, we shall run the K fold cross-validation for the models with different values of\nn_neighbors\n, as shown below.\n# search for an optimal value of K for KNN\nk_range\n=\nlist\n(\nrange\n(\n1\n,\n31\n))\nk_scores\n=\n[]\nfor\nk\nin\nk_range\n:\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\nk\n)\nscores\n=\ncross_val_score\n(\nknn\n,\nX\n,\ny\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n)\nk_scores\n.\nappend\n(\nscores\n.\nmean\n())\nprint\n(\nk_scores\n)\n# Output k_scores\n[\n0.96\n,\n0.9533333333333334\n,\n0.9666666666666666\n,\n0.9666666666666666\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9733333333333334\n,\n0.9666666666666668\n,\n0.9666666666666668\n,\n0.9733333333333334\n,\n0.9800000000000001\n,\n0.9733333333333334\n,\n0.9733333333333334\n,\n0.9733333333333334\n,\n0.9733333333333334\n,\n0.9800000000000001\n,\n0.9733333333333334\n,\n0.9800000000000001\n,\n0.9666666666666666\n,\n0.9666666666666666\n,\n0.9733333333333334\n,\n0.96\n,\n0.9666666666666666\n,\n0.96\n,\n0.9666666666666666\n,\n0.9533333333333334\n,\n0.9533333333333334\n,\n0.9533333333333334\n]\nEnter fullscreen mode\nExit fullscreen mode\nLet us plot the values to get a better idea.\nimport\nmatplotlib.pyplot\nas\nplt\n%\nmatplotlib\ninline\n# plot the value of K for KNN (x-axis) versus the cross-validated accuracy (y-axis)\nplt\n.\nplot\n(\nk_range\n,\nk_scores\n)\nplt\n.\nxlabel\n(\n'\nValue of K for KNN\n'\n)\nplt\n.\nylabel\n(\n'\nCross-Validated Accuracy\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\nWe see that\nn_neighbors (K)\nvalues from 13 to 20 yield higher accuracy, especially\nK\n=13,18 and 20.As a larger value of\nK\nyields a less complex model, we choose\nK\n=20.\nThis process of searching for the optimal values of hyperparameters is called\nhyperparameter tuning\n.\nIn this example, we chose the values of\nK\nthat resulted in higher mean accuracy score under 10-fold cross validation.\nThis is how cross-validation can be used to search for the best hyperparameters and this can be done much more efficiently in scikit-learn.\nIn KNN classifiers, setting a very small value for\nK\nwill make the model needlessly complex, and a very large value of\nK\nwould result in a model with high bias that yields suboptimal performance.\nAs\nK\n=13,18 and 20 gave the highest accuracy score, close to 0.98, we decided to choose\nK\n=20 as a larger value of\nK\nwould yield a less complex model.\nWhile it is not particularly difficult to write the for loop, we do realize that we may have to do it often.\nTherefore, it would be good to have a more convenient way of doing the hyperparameter search, without having to write a loop every time and identify the best parameter through inspection.\nGrid Search class in scikit-learn helps us do this all the more easily. Let's learn about it in the next section.\nHow to Use Grid Search for Hyperparameter Search\n▶\nImport GridSearchCV Class\nfrom\nsklearn.model_selection\nimport\nGridSearchCV\nEnter fullscreen mode\nExit fullscreen mode\n▶\nDefine the Parameter Grid\nWe now define the\nparameter grid\n(\nparam_grid\n), a Python dictionary, whose key is the name of the hyperparameter whose best value we’re trying to find and the value is the list of possible values that we would like to search over for the hyperparameter.\n# define the parameter values that should be searched\nk_range\n=\nlist\n(\nrange\n(\n1\n,\n31\n))\n# create a parameter grid: map the parameter names to the values that should be searched\nparam_grid\n=\ndict\n(\nn_neighbors\n=\nk_range\n)\nprint\n(\nparam_grid\n)\n# param_grid\n{\n'\nn_neighbors\n'\n:\n[\n1\n,\n2\n,\n3\n,\n4\n,\n5\n,\n6\n,\n7\n,\n8\n,\n9\n,\n10\n,\n11\n,\n12\n,\n13\n,\n14\n,\n15\n,\n16\n,\n17\n,\n18\n,\n19\n,\n20\n,\n21\n,\n22\n,\n23\n,\n24\n,\n25\n,\n26\n,\n27\n,\n28\n,\n29\n,\n30\n]}\nEnter fullscreen mode\nExit fullscreen mode\nWe now instantiate\nGridSearchCV\n. Note that we specify the\nparam_grid\ninstead of the\nn_neighbors\nargument that we had specified for\ncross_val_score\nearlier.\nWhy is this valid?\nRemember, the parameter grid,\nparam_grid\nis a dictionary whose key is\nn_neighbors\nand the value is a list of possible values of\nn_neighbors\n. Therefore, specifying the\nparam_grid\nensures that the value at index\ni\nis fetched as the value of\nn_neighbors\nin the i_th run.\n▶\nInstantiate, fit the grid and view the results\n# instantiate the grid\ngrid\n=\nGridSearchCV\n(\nknn\n,\nparam_grid\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nreturn_train_score\n=\nFalse\n)\nEnter fullscreen mode\nExit fullscreen mode\nWe now go ahead and fit the grid with data, and access the\ncv_results_\nattribute to get the mean accuracy score after 10-fold cross-validation, standard deviation and the parameter values.\nFor convenience, we may store the results in a pandas DataFrame. The mean and standard deviation of the accuracy scores for\nn_neighbors\n=1 to 10 are shown below.\n# fit the grid with data\ngrid\n.\nfit\n(\nX\n,\ny\n)\n# view the results as a pandas DataFrame\nimport\npandas\nas\npd\npd\n.\nDataFrame\n(\ngrid\n.\ncv_results_\n)[[\n'\nmean_test_score\n'\n,\n'\nstd_test_score\n'\n,\n'\nparams\n'\n]]\n# Output\nmean_test_score\nstd_test_score\nparams\n0\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n1\n}\n1\n0.953333\n0.052068\n{\n'\nn_neighbors\n'\n:\n2\n}\n2\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n3\n}\n3\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n4\n}\n4\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n5\n}\n5\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n6\n}\n6\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n7\n}\n7\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n8\n}\n8\n0.973333\n0.032660\n{\n'\nn_neighbors\n'\n:\n9\n}\n9\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n10\n}\nEnter fullscreen mode\nExit fullscreen mode\nWhen using\ncross_val_score\n, we tried eyeballing the accuracy scores to identify the best hyperparameters and to make it easier, we plotted the value of hyperparameters vs the respective cross-validated accuracy scores!\nSounds good but doesn’t seem to be a great option though!\nOnce we’ve completed the grid search, the following attributes can be very useful! We can choose to examine:\n☑ the\nbest_score_\n, the\nhighest cross-validated accuracy score\n☑ the\nbest_params_\n, the\noptimal value for the hyperparameters\n, and\n☑ the\nbest_estimator_\n, which is the\nbest model\nthat has the\nbest hyperparameter\n.\nLet's now examine these for our example.\n▶\nExamine the best score and best hyperparameters\n# examine the best model\nprint\n(\ngrid\n.\nbest_score_\n)\nprint\n(\ngrid\n.\nbest_params_\n)\nprint\n(\ngrid\n.\nbest_estimator_\n)\n# Output\n0.9800000000000001\n{\n'\nn_neighbors\n'\n:\n13\n}\nKNeighborsClassifier\n(\nalgorithm\n=\n'\nauto\n'\n,\nleaf_size\n=\n30\n,\nmetric\n=\n'\nminkowski\n'\n,\nmetric_params\n=\nNone\n,\nn_jobs\n=\nNone\n,\nn_neighbors\n=\n13\n,\np\n=\n2\n,\nweights\n=\n'\nuniform\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\nK\n=13 has been chosen, remember,\nK\n=13 was one of the values of\nK\nthat gave highest cross-validated accuracy score.✔\nSo far so good!\nSearching for Multiple Hyperparameters\nIn this example, the only hyperparameter that we searched for was\nn_neighbors\n.\nWhat if there were many such hyperparameters?\nWe may think,\n“Why not tune each hyperparameter independently?”\nWell, we may independently search for the optimal values for each of the hyperparameters; but\nthe model may perform best at some values of the parameters that are very different from the individual best values\n.\nSo, we have to search for the\ncombination of the parameters\nthat\noptimizes performance\nrather than the individual best parameters.\nLet's build on the same example of KNNClassifier.\nIn addition to\nn_neighbors\n, let's search for the optimal weighting strategy as well.\nThe default weighting option is\n‘uniform’\nwhere all points are weighted equally and\n‘distance’\noption weights points by the inverse of their distance.\nIn this case,\ncloser neighbors\nof a query point will have a\ngreater influence\nthan neighbors which are far away.\n▶\nDefine the Parameter Grid\n# define the parameter values that should be searched\nk_range\n=\nlist\n(\nrange\n(\n1\n,\n31\n))\nweight_options\n=\n[\n'\nuniform\n'\n,\n'\ndistance\n'\n]\n# create a parameter grid: map the parameter names to the values that should be searched\nparam_grid\n=\ndict\n(\nn_neighbors\n=\nk_range\n,\nweights\n=\nweight_options\n)\nprint\n(\nparam_grid\n)\n# param_grid\n{\n'\nn_neighbors\n'\n:\n[\n1\n,\n2\n,\n3\n,\n4\n,\n5\n,\n6\n,\n7\n,\n8\n,\n9\n,\n10\n,\n11\n,\n12\n,\n13\n,\n14\n,\n15\n,\n16\n,\n17\n,\n18\n,\n19\n,\n20\n,\n21\n,\n22\n,\n23\n,\n24\n,\n25\n,\n26\n,\n27\n,\n28\n,\n29\n,\n30\n],\n'\nweights\n'\n:\n[\n'\nuniform\n'\n,\n'\ndistance\n'\n]}\nEnter fullscreen mode\nExit fullscreen mode\nLet us instantiate and fit the grid and view results, as before.\n# instantiate and fit the grid\ngrid\n=\nGridSearchCV\n(\nknn\n,\nparam_grid\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nreturn_train_score\n=\nFalse\n)\ngrid\n.\nfit\n(\nX\n,\ny\n)\n# view the results\npd\n.\nDataFrame\n(\ngrid\n.\ncv_results_\n)[[\n'\nmean_test_score\n'\n,\n'\nstd_test_score\n'\n,\n'\nparams\n'\n]]\n# Results\nmean_test_score\nstd_test_score\nparams\n0\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n1\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n1\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n1\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n2\n0.953333\n0.052068\n{\n'\nn_neighbors\n'\n:\n2\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n3\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n2\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n4\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n3\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n5\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n3\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n6\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n4\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n7\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n4\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n8\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n5\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n9\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n5\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n10\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n6\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n11\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n6\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n12\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n7\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n13\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n7\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n14\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n8\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n15\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n8\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n16\n0.973333\n0.032660\n{\n'\nn_neighbors\n'\n:\n9\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n17\n0.973333\n0.032660\n{\n'\nn_neighbors\n'\n:\n9\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n18\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n10\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\nEnter fullscreen mode\nExit fullscreen mode\nWe see that we have 30*2=60 models.(As we had 30 possible values for\nn_neighbors\nand 2 possible values for weights)\nAs we chose 10-fold cross-validation, there will be 60*10=600 predictions made!\nTime to look at our model’s best score and parameters that yielded the best score.\n▶\nExamine the best score and best hyperparameters\n# examine the best model\nprint\n(\ngrid\n.\nbest_score_\n)\nprint\n(\ngrid\n.\nbest_params_\n)\n# best score and best parameters\n0.9800000000000001\n{\n'\nn_neighbors\n'\n:\n13\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\nEnter fullscreen mode\nExit fullscreen mode\nWe obtain the same best cross-validated accuracy score of 0.98, with\nn_neighbors\n=13 and\nweights\n= ‘uniform’.\nNow, let's suppose we have to tune 4 hyperparameters and we have a list of 10 possible values for each of the hyperparameters.\nThis process creates 10*10*10*10 =10,000 models and when we run 10 fold cross-validation, there are 100,000 predictions made.\nClearly, things do scale up very quickly and can soon become computationally infeasible.\nHow to Search for Hyperparameters More Efficiently Using Randomized Search\nWe shall now try to rephrase the limitations of Grid Search better, in a more formal way.\nSay we have to search for\nM\nparameters; Let\np_1, p_2,p_3, …, p_M\nbe the\nM\nparameters.\nLet the number of values that we would like to search over for\np_1\nbe\nn1\n, for\np_2\nbe\nn2\n, and so on, with\nnM\nvalues for\np_M\n.\nGrid Search considers\nall possible hyperparameter settings\n(combinations) into account and creates a model for each possible setting to choose the best model with optimal hyperparameters.\nTo understand it better, assume that out of\nM\nparameters, we decide to freeze the values of all hyperparameters except one, say the M_th parameter\np_M\n. So, Grid Search involves searching through the list of\nnM\nvalues for the M_th hyperparameter; And\nnM\nmodels are created.\nSuppose we now freeze the values of all hyperparameters except two, say the last two (\np_M\nand\np_(M-1)\n). We now have to search through all possible combinations of\np_M\nand\np_(M-1)\n, each having\nnM\nand\nn_(M-1)\npossible values that we could search over.\nWe now take a step back and freeze the value of\np_M-1\nand search through all values for\np_M\n; To account for all possible combinations, we should repeat the procedure for all\nn_M-1\nvalues for\np_M-1\n. So, this process would leave us with\nn_(M-1) * nM\nmodels.\nHope it’s clear how the complexity scales with increase in the number of values each hyperparameter could take.\nFor the above example with\nM\nhyperparameters, we would have\nn1*n2*n3*…*n_M\nmodels. This is why we said that things could scale up quickly and become computationally intractable with Grid Search.\nWith this motivation to make hyperparameter search computationally more efficient, let us proceed to understand Randomized Search.\nUnderstanding RandomizedSearchCV\nIn contrast to\nGridSearchCV\n, not all parameter values are tried out in\nRandomizedSearchCV\n, but rather a fixed number of parameter settings is sampled from the specified distributions/ list of parameters.\nIf some of the hyperparameters that we’re searching for are continuous, then we should specify the distribution rather than the list of values, while defining the parameter grid. How do we define the fixed number of parameter settings?\nThe number of parameter settings that are tried is given by\nn_iter\n. There's a quality vs computational cost trade-off in picking\nn_iter\n.\nA very small value of\nn_iter\nwould imply that we’re more likely to find a suboptimal solution, because we are actually considering too few combinations.\nA very high value of n_iter would mean we can ideally get closer to finding the best hyperparameters that yield the best model, but this again comes with a high computation cost as before.\nIn fact, if we set\nn_iter= n1*n2*n3*…*n_M\nfrom the previous example, then, we’re essentially considering all possible hyperparameter combinations and now Randomized Search and Grid Search are equivalent.\nLet us build on the same example of KNNClassifier from the previous section.\nLet us search for the optimal weighting strategy and\nn_neighbors\n. And now, let us implement Randomized Search in scikit-learn and do the following steps, as we did for Grid Search.\n▶\nImport RandomizedSearchCV class\nfrom\nsklearn.model_selection\nimport\nRandomizedSearchCV\nEnter fullscreen mode\nExit fullscreen mode\n▶\nDefine the parameter grid\n# specify \"parameter distributions\" rather than a \"parameter grid\"\nparam_dist\n=\ndict\n(\nn_neighbors\n=\nk_range\n,\nweights\n=\nweight_options\n)\nEnter fullscreen mode\nExit fullscreen mode\n▶\nInstantiate the grid; Set n_iter=10, Fit the grid & View the results\n# n_iter controls the number of searches\nrand\n=\nRandomizedSearchCV\n(\nknn\n,\nparam_dist\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nn_iter\n=\n10\n,\nrandom_state\n=\n5\n,\nreturn_train_score\n=\nFalse\n)\nrand\n.\nfit\n(\nX\n,\ny\n)\npd\n.\nDataFrame\n(\nrand\n.\ncv_results_\n)[[\n'\nmean_test_score\n'\n,\n'\nstd_test_score\n'\n,\n'\nparams\n'\n]]\n#DataFrame\nmean_test_score\nstd_test_score\nparams\n0\n0.973333\n0.032660\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n16\n}\n1\n0.966667\n0.033333\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n22\n}\n2\n0.980000\n0.030551\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n18\n}\n3\n0.966667\n0.044721\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n27\n}\n4\n0.953333\n0.042687\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n29\n}\n5\n0.973333\n0.032660\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n10\n}\n6\n0.966667\n0.044721\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n22\n}\n7\n0.973333\n0.044222\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n14\n}\n8\n0.973333\n0.044222\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n12\n}\n9\n0.973333\n0.032660\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n15\n}\nEnter fullscreen mode\nExit fullscreen mode\n▶\nExamine the best score and best hyperparameters\n# examine the best model\nprint\n(\nrand\n.\nbest_score_\n)\nprint\n(\nrand\n.\nbest_params_\n)\n# Output\n0.9800000000000001\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n18\n}\nEnter fullscreen mode\nExit fullscreen mode\n▶\nParameters of the best model\nSurprisingly, we see that the highest accuracy score obtained in this case, where we only looked at 10 different parameter settings instead of 60 in Grid Search, is the same as before: 0.98 ✔\nAnd the value for\nn_neighbors= 18\n, which is also one of the optimal values that we got when we initially searched for the optimal value of\nn_neighbors\n.\nMaybe we just got lucky?\nWhat is the guarantee that we will always get the best results?\nAh, this question makes perfect sense, doesn’t it?\nLet us do the following now: Let us run\nRandomizedSearchCV\nfor multiple times and see how many times we really end up getting lucky!\n➡️Run RandomizedSearchCV 20 times and see what happens; We log the best_score_ for every run.\n# run RandomizedSearchCV 20 times (with n_iter=10) and record the best score\nbest_scores\n=\n[]\nfor\n_\nin\nrange\n(\n20\n):\nrand\n=\nRandomizedSearchCV\n(\nknn\n,\nparam_dist\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nn_iter\n=\n10\n,\nreturn_train_score\n=\nFalse\n)\nrand\n.\nfit\n(\nX\n,\ny\n)\nbest_scores\n.\nappend\n(\nround\n(\nrand\n.\nbest_score_\n,\n3\n))\nLet\nus\nexamine\nall\nthe\n20\nbest\nscores\nnow\n.\nprint\n(\nbest_scores\n)\n# Output: Best Scores\n[\n0.973\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.973\n,\n0.98\n,\n0.98\n,\n0.973\n,\n0.98\n,\n0.973\n,\n0.973\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.973\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.973\n]\nEnter fullscreen mode\nExit fullscreen mode\nUpon examining the best scores above for all the 20 runs, we see that we get the best accuracy score of 0.98 about 13 times.\nLooks like we’re lucky indeed! What about the other 7 times when we didn't quite get the best accuracy score? These accuracy scores are around 0.973 which is pretty close to 0.98.\nThis observation convinces us that even though Randomized Search may not always give the hyperparameters of the best performing model, the models obtained by using these hyperparameters\ndo not perform much worse compared to the best model\nobtained from Grid Search.\nThis means, the best models thus obtained, with the hyperparameters from randomized search are clearly very close to the optimal model.\nIn essence, these\nmay not be the best hyperparameters\n, but certainly close to the best hyperparameters, except that these are found under resource-constrained settings.\nHope you all understood how we could use Randomized Search for hyperparameter tuning.\nHope you all enjoyed reading this post.😀\nHappy Learning! Until next time!✨\nReferences\n[1]\nhttps://courses.dataschool.io/courses/introduction-to-machine-learning-with-scikit-learn\n[2]\nhttp://scikitlearn.org/stable/modules/cross_validation.html\n[3]\nhttp://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\n[4]\nhttp://scikitlearn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html\nCover Image: Photo by\nRoss Sneddon\non\nUnsplash\nNote\n: This post is a comprehensive post for understanding model evaluation, and hyperparameter search. This is a compiled, and improved version of a few of my earlier shorter posts.",
  "metadata": {
    "title": "Cross-Validation and Hyperparameter Search in Scikit-Learn - A Complete Guide - DEV Community",
    "description": "This post is inspired by Kevin Markham's course Introduction to Machine Learning with scikit-learn on... Tagged with python, tutorial, machinelearning.",
    "keywords": "python, tutorial, machinelearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/cross-validation-and-hyperparameter-search-in-scikit-learn-a-complete-guide-5ed8",
    "og:title": "Cross-Validation and Hyperparameter Search in Scikit-Learn - A Complete Guide",
    "og:description": "This post is inspired by Kevin Markham's course Introduction to Machine Learning with scikit-learn on...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fznynx244hrwqj0h3f5i2.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Cross-Validation and Hyperparameter Search in Scikit-Learn - A Complete Guide",
    "twitter:description": "This post is inspired by Kevin Markham's course Introduction to Machine Learning with scikit-learn on...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fznynx244hrwqj0h3f5i2.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/6c5a439f68cd/code/body-000.txt",
      "chars": 171,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-001.txt",
      "chars": 113,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-002.txt",
      "chars": 134,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-003.txt",
      "chars": 172,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-004.txt",
      "chars": 1026,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-005.txt",
      "chars": 482,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-006.txt",
      "chars": 851,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-007.txt",
      "chars": 236,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-008.txt",
      "chars": 49,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-009.txt",
      "chars": 371,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-010.txt",
      "chars": 113,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-011.txt",
      "chars": 701,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-012.txt",
      "chars": 341,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-013.txt",
      "chars": 473,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-014.txt",
      "chars": 1606,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-015.txt",
      "chars": 168,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-016.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-017.txt",
      "chars": 130,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-018.txt",
      "chars": 1015,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-019.txt",
      "chars": 145,
      "language": "text"
    },
    {
      "file": "attachments/6c5a439f68cd/code/body-020.txt",
      "chars": 495,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwdzir812ln0cbv4jqsqx.png",
      "bytes": 5681,
      "reused": true,
      "local_path": "attachments/6c5a439f68cd/images/002.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwdzir812ln0cbv4jqsqx.png",
      "alt": "image",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://courses.dataschool.io/courses/introduction-to-machine-learning-with-scikit-learn",
      "label": "Introduction to Machine Learning with scikit-learn",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fwdzir812ln0cbv4jqsqx.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "http://scikitlearn.org/stable/modules/cross_validation.html",
      "label": "http://scikitlearn.org/stable/modules/cross_validation.html",
      "type": "a"
    },
    {
      "href": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html",
      "label": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html",
      "type": "a"
    },
    {
      "href": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html",
      "label": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@rosssneddon?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Ross Sneddon",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:09Z",
  "source_url": "https://dev.to/balapriya/cross-validation-and-hyperparameter-search-in-scikit-learn-a-complete-guide-5ed8",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 27906,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6c5a439f68cd/pages/source.html",
  "local_html": "attachments/6c5a439f68cd/pages/article.html",
  "body_base_path": "attachments/6c5a439f68cd/pages/",
  "body_markdown_path": "attachments/6c5a439f68cd/pages/article.md"
}