{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "Hyperparameter Tuning: Understanding Grid Search",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/hyperparameter-tuning-understanding-grid-search-2648",
  "publish_date": "2021-04-14",
  "corpus_stage": "",
  "notes": "",
  "id": "7f7f179e73ac",
  "collected_at": "2026-10-02T14:01:45",
  "title": "Hyperparameter Tuning: Understanding Grid Search - DEV Community",
  "body_markdown": "In the previous blog post,\nRole of Cross-Validation\n, we had looked at how train/test split doesn’t suffice to get a reliable estimate of the out-of-sample accuracy which motivated us to understand the importance of cross-validation.\nIn this blog post, we shall see how we can do hyperparameter search more conveniently with\nGridSearchCV\nin scikit-learn, and learn how to use it to tune multiple hyperparameters.Let’s get started!\nTime for a quick recall!\nWe were looking at classification problem on the iris dataset; We had used 10-fold cross-validation and the mean value of the cross-validated accuracy scores as the estimate of out-of-sample accuracy.☑\nLet’s quickly walk through the steps again.\n# necessary imports\nfrom\nsklearn.datasets\nimport\nload_iris\nfrom\nsklearn.neighbors\nimport\nKNeighborsClassifier\nfrom\nsklearn.model_selection\nimport\ncross_val_score\nimport\nmatplotlib.pyplot\nas\nplt\n%\nmatplotlib\ninline\nEnter fullscreen mode\nExit fullscreen mode\n# read in the iris data\niris\n=\nload_iris\n()\n# create X (features) and y (response)\nX\n=\niris\n.\ndata\ny\n=\niris\n.\ntarget\nEnter fullscreen mode\nExit fullscreen mode\n# 10-fold cross-validation with K=5 for KNN (the n_neighbors parameter)\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\n5\n)\nscores\n=\ncross_val_score\n(\nknn\n,\nX\n,\ny\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n)\n# use average accuracy as an estimate of out-of-sample accuracy\nprint\n(\nscores\n.\nmean\n())\n# Output\n0.9666666666666668\nEnter fullscreen mode\nExit fullscreen mode\nWe then wanted to identify the best value of K for the KNN classifier. To find the best\nK\n, we looped through a possible set of values for\nK\n(\nn_neighbors\n) and then picked the value of\nK\nthat gave the highest cross-validated accuracy.\nIn KNN classifier, setting a\nvery small\nvalue for\nK\nwill make the model\nneedlessly complex\n, and a\nvery large\nvalue of\nK\nwould result in a model with high bias, that yields suboptimal performance.\n# search for an optimal value of K for KNN\nk_range\n=\nlist\n(\nrange\n(\n1\n,\n31\n))\nk_scores\n=\n[]\nfor\nk\nin\nk_range\n:\nknn\n=\nKNeighborsClassifier\n(\nn_neighbors\n=\nk\n)\nscores\n=\ncross_val_score\n(\nknn\n,\nX\n,\ny\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n)\nk_scores\n.\nappend\n(\nscores\n.\nmean\n())\n# plot the value of K for KNN (x-axis) versus the cross-validated accuracy (y-axis)\nplt\n.\nplot\n(\nk_range\n,\nk_scores\n)\nplt\n.\nxlabel\n(\n'\nValue of K for KNN\n'\n)\nplt\n.\nylabel\n(\n'\nCross-Validated Accuracy\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\nAs K=13,18 and 20 gave the highest accuracy score, close to 0.98, we decided to choose K=20 as a larger value of K would yield a less complex model. While it is not particularly difficult to write the for loop, we do realize that we may have to do it often.\nTherefore, it would be good to have a more convenient way of doing the hyperparameter search, without having to write a loop every time and identify the best parameter through inspection.\nUnderstanding GridSearchCV\nLet’s go ahead and import the GridSearchCV class.\nfrom\nsklearn.model_selection\nimport\nGridSearchCV\nEnter fullscreen mode\nExit fullscreen mode\nDefine the Parameter Grid\nWe now define the parameter grid (\nparam_grid\n), a Python dictionary, whose key is the name of the hyperparameter whose best value we’re trying to find and the value is the list of possible values that we would like to search over for the hyperparameter.\n# define the parameter values that should be searched\nk_range\n=\nlist\n(\nrange\n(\n1\n,\n31\n))\n# create a parameter grid: map the parameter names to the values that should be searched\nparam_grid\n=\ndict\n(\nn_neighbors\n=\nk_range\n)\nprint\n(\nparam_grid\n)\n# param_grid\n{\n'\nn_neighbors\n'\n:\n[\n1\n,\n2\n,\n3\n,\n4\n,\n5\n,\n6\n,\n7\n,\n8\n,\n9\n,\n10\n,\n11\n,\n12\n,\n13\n,\n14\n,\n15\n,\n16\n,\n17\n,\n18\n,\n19\n,\n20\n,\n21\n,\n22\n,\n23\n,\n24\n,\n25\n,\n26\n,\n27\n,\n28\n,\n29\n,\n30\n]}\nEnter fullscreen mode\nExit fullscreen mode\nWe now instantiate\nGridSearchCV\n. Note that we specify the\nparam_grid\ninstead of\nn_neighbors\nargument that we had specified for\ncross_val_score\nearlier. Why is this valid?\nRemember, the parameter grid,\nparam_grid\nis a dictionary whose key is\nn_neighbors\nand the value is a list of possible values of n_neighbors. Therefore, specifying the\nparam_grid\nensures that the value at index\ni\nis fetched as the value of\nn_neighbors\nin the\ni_th\nrun.\nInstantiate, fit grid and view results\n# instantiate the grid\ngrid\n=\nGridSearchCV\n(\nknn\n,\nparam_grid\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nreturn_train_score\n=\nFalse\n)\nEnter fullscreen mode\nExit fullscreen mode\nWe now go ahead and fit the grid with data, and access the\ncv_results_\nattribute to get the mean accuracy score after 10-fold cross-validation, standard deviation and the parameter values. For convenience, we may store the results in a pandas DataFrame. The mean and standard deviation of the accuracy scores for n_neighbors =1 to 10 are shown below.\n# fit the grid with data\ngrid\n.\nfit\n(\nX\n,\ny\n)\n# view the results as a pandas DataFrame\nimport\npandas\nas\npd\npd\n.\nDataFrame\n(\ngrid\n.\ncv_results_\n)[[\n'\nmean_test_score\n'\n,\n'\nstd_test_score\n'\n,\n'\nparams\n'\n]]\n# Output\nmean_test_score\nstd_test_score\nparams\n0\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n1\n}\n1\n0.953333\n0.052068\n{\n'\nn_neighbors\n'\n:\n2\n}\n2\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n3\n}\n3\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n4\n}\n4\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n5\n}\n5\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n6\n}\n6\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n7\n}\n7\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n8\n}\n8\n0.973333\n0.032660\n{\n'\nn_neighbors\n'\n:\n9\n}\n9\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n10\n}\nEnter fullscreen mode\nExit fullscreen mode\nWhen using\ncross_val_score\n, we tried eyeballing the accuracy scores to identify the best hyperparameters and to make it easier, we plotted the value of hyperparameters vs the respective cross-validated accuracy scores!\nSounds good but doesn’t seem to be a great option!\nOnce we’ve completed the grid search, the following attributes can be very useful!\nWe can choose to examine:\n☑ the\nbest_score_\n, the highest cross-validated accuracy score\n☑ the\nbest_params_\n, the optimal value for the hyperparameters, and\n☑ the\nbest_estimator_\n, which is the best model that has the best hyperparameter.\nLet us now examine these for our example.\n# examine the best model\nprint\n(\ngrid\n.\nbest_score_\n)\nprint\n(\ngrid\n.\nbest_params_\n)\nprint\n(\ngrid\n.\nbest_estimator_\n)\n# Output\n0.9800000000000001\n{\n'\nn_neighbors\n'\n:\n13\n}\nKNeighborsClassifier\n(\nalgorithm\n=\n'\nauto\n'\n,\nleaf_size\n=\n30\n,\nmetric\n=\n'\nminkowski\n'\n,\nmetric_params\n=\nNone\n,\nn_jobs\n=\nNone\n,\nn_neighbors\n=\n13\n,\np\n=\n2\n,\nweights\n=\n'\nuniform\n'\n)\nEnter fullscreen mode\nExit fullscreen mode\nK=13 has been chosen, remember, K=13 was one of the values of K that gave highest cross-validated accuracy score.✔\nSo far so good!\nSearching for multiple parameters\nIn this example, the only hyperparameter that we searched for was\nn_neighbors\n. What if there were many such hyperparameters?\nWe may think,\n“Why not tune each hyperparameter independently?”\nWell, we may independently search for the optimal values for each of the hyperparameters; but the model may perform best at some values of the parameters that are very different from the individual best values. So, we have to search for the combination of the parameters that optimizes performance rather than the individual best parameters.\nLet us build on the same example of\nKNNClassifier\n, in addition to\nn_neighbors\n, let us also search for the optimal weighting strategy.\nThe default weighting option is ‘uniform’ where all points are weighted equally and ‘distance’ option weights points by the inverse of their distance. In this case, closer neighbors of a query point will have a greater influence than neighbors which are further away.\n# define the parameter values that should be searched\nk_range\n=\nlist\n(\nrange\n(\n1\n,\n31\n))\nweight_options\n=\n[\n'\nuniform\n'\n,\n'\ndistance\n'\n]\n# create a parameter grid: map the parameter names to the values that should be searched\nparam_grid\n=\ndict\n(\nn_neighbors\n=\nk_range\n,\nweights\n=\nweight_options\n)\nprint\n(\nparam_grid\n)\n# param_grid\n{\n'\nn_neighbors\n'\n:\n[\n1\n,\n2\n,\n3\n,\n4\n,\n5\n,\n6\n,\n7\n,\n8\n,\n9\n,\n10\n,\n11\n,\n12\n,\n13\n,\n14\n,\n15\n,\n16\n,\n17\n,\n18\n,\n19\n,\n20\n,\n21\n,\n22\n,\n23\n,\n24\n,\n25\n,\n26\n,\n27\n,\n28\n,\n29\n,\n30\n],\n'\nweights\n'\n:\n[\n'\nuniform\n'\n,\n'\ndistance\n'\n]}\nEnter fullscreen mode\nExit fullscreen mode\nLet us instantiate and fit grid and view results, as before.\n# instantiate and fit the grid\ngrid\n=\nGridSearchCV\n(\nknn\n,\nparam_grid\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nreturn_train_score\n=\nFalse\n)\ngrid\n.\nfit\n(\nX\n,\ny\n)\n# view the results\npd\n.\nDataFrame\n(\ngrid\n.\ncv_results_\n)[[\n'\nmean_test_score\n'\n,\n'\nstd_test_score\n'\n,\n'\nparams\n'\n]]\n# Results\nmean_test_score\nstd_test_score\nparams\n0\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n1\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n1\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n1\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n2\n0.953333\n0.052068\n{\n'\nn_neighbors\n'\n:\n2\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n3\n0.960000\n0.053333\n{\n'\nn_neighbors\n'\n:\n2\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n4\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n3\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n5\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n3\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n6\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n4\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n7\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n4\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n8\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n5\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n9\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n5\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n10\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n6\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n11\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n6\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n12\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n7\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n13\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n7\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n14\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n8\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n15\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n8\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n16\n0.973333\n0.032660\n{\n'\nn_neighbors\n'\n:\n9\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\n17\n0.973333\n0.032660\n{\n'\nn_neighbors\n'\n:\n9\n,\n'\nweights\n'\n:\n'\ndistance\n'\n}\n18\n0.966667\n0.044721\n{\n'\nn_neighbors\n'\n:\n10\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\nEnter fullscreen mode\nExit fullscreen mode\nResults of Grid Search for two parameters (Truncated View of the DataFrame)\nOnly the first ten rows of the results have been shown above; We actually have 30*2=60 models (as we had 30 possible values for\nn_neighbors\nand 2 possible values for weights) and as we chose 10-fold cross-validation, there will be 60*10=600 predictions made! Time to look at our model’s best score and parameters that yielded the best score.\n# examine the best model\nprint\n(\ngrid\n.\nbest_score_\n)\nprint\n(\ngrid\n.\nbest_params_\n)\n# best score and best parameters\n0.9800000000000001\n{\n'\nn_neighbors\n'\n:\n13\n,\n'\nweights\n'\n:\n'\nuniform\n'\n}\nEnter fullscreen mode\nExit fullscreen mode\nWe obtain the same best cross-validated accuracy score of 0.98, with\nn_neighbors\n=13 and\nweights\n= ‘uniform’. Now, let us say we have to tune 4 hyperparameters and we have a list of 10 possible values for each of the hyperparameters. This process creates\n10*10*10*10 =10,000\nmodels and when we run 10 fold cross-validation, there are 100,000 predictions made. Clearly, things do scale up very quickly and can soon become computationally infeasible.\nMore efficient hyperparameter searches such as Randomized Search and Informed Search can be very useful in overcoming this drawback. Let us cover them in a subsequent blog post.\nUntil then, Happy Learning!\nReferences\n[1] Here’s the\nlink\nto the Google Colab notebook for the above used example.\n[2]\nIntroduction to Machine Learning in Python with scikit-learn by DataSchool\n.\n[3] Scikit-learn documentation: GridSearchCV\nhttp://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html\nCover Image: Photo by\nKelly Sikkema\non\nUnsplash",
  "metadata": {
    "title": "Hyperparameter Tuning: Understanding Grid Search - DEV Community",
    "description": "In the previous blog post, Role of Cross-Validation, we had looked at how train/test split doesn’t su... Tagged with beginners, machinelearning, datascience, python.",
    "keywords": "beginners, machinelearning, datascience, python, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/hyperparameter-tuning-understanding-grid-search-2648",
    "og:title": "Hyperparameter Tuning: Understanding Grid Search",
    "og:description": "In the previous blog post, Role of Cross-Validation, we had looked at how train/test split doesn’t su...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fc5zk3xs6e5tj0ascpc1s.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Hyperparameter Tuning: Understanding Grid Search",
    "twitter:description": "In the previous blog post, Role of Cross-Validation, we had looked at how train/test split doesn’t su...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fc5zk3xs6e5tj0ascpc1s.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/7f7f179e73ac/code/body-000.txt",
      "chars": 213,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-001.txt",
      "chars": 113,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-002.txt",
      "chars": 290,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-003.txt",
      "chars": 434,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-004.txt",
      "chars": 49,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-005.txt",
      "chars": 373,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-006.txt",
      "chars": 113,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-007.txt",
      "chars": 702,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-008.txt",
      "chars": 341,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-009.txt",
      "chars": 474,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-010.txt",
      "chars": 1606,
      "language": "text"
    },
    {
      "file": "attachments/7f7f179e73ac/code/body-011.txt",
      "chars": 169,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F28homaa89irt6v1ezq4c.png",
      "bytes": 16324,
      "reused": true,
      "local_path": "attachments/7f7f179e73ac/images/002.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F28homaa89irt6v1ezq4c.png",
      "alt": "Alt Text",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://dev.to/balapriya/role-of-cross-validation-in-model-validation-and-hyperparameter-search-n3",
      "label": "Role of Cross-Validation",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F28homaa89irt6v1ezq4c.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1VUv1bVxBsyLT9CcXZq5J0KMZB8p44WSh?usp=sharing",
      "label": "link",
      "type": "a"
    },
    {
      "href": "https://www.dataschool.io/machine-learning-with-scikit-learn/",
      "label": "Introduction to Machine Learning in Python with scikit-learn by DataSchool",
      "type": "a"
    },
    {
      "href": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html",
      "label": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@kellysikkema?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Kelly Sikkema",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/s/photos/grid-books?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:12Z",
  "source_url": "https://dev.to/balapriya/hyperparameter-tuning-understanding-grid-search-2648",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11701,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/7f7f179e73ac/pages/source.html",
  "local_html": "attachments/7f7f179e73ac/pages/article.html",
  "body_base_path": "attachments/7f7f179e73ac/pages/",
  "body_markdown_path": "attachments/7f7f179e73ac/pages/article.md"
}