{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "Hyperparameter Tuning: Understanding Randomized Search",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/hyperparameter-tuning-understanding-randomized-search-343l",
  "publish_date": "2021-04-19",
  "corpus_stage": "",
  "notes": "",
  "id": "28bc1d271498",
  "collected_at": "2026-10-02T14:01:45",
  "title": "Hyperparameter Tuning: Understanding Randomized Search - DEV Community",
  "body_markdown": "Hi everyone!\nThis is the third of the series of articles on cross-validation and hyperparameter tuning. Here are the links to the first two if you’d like to give them a read before going ahead with reading this blog post.\nRole of Cross-Validation in Model Validation and Hyperparameter Search\nHyperparameter Tuning: Understanding Grid Search\nLet us quickly outline the key ideas that we’ve covered thus far.\nIn the first article, we observed how\ntrain/test split\ndepends heavily on the particular data records that end up in the training and test sets, which led us to understand the need for cross-validation as a way to effectively evaluate a model’s performance.\nWe then saw how cross-validation can be used in hyperparameter search; If you can remember, we decided to choose that value of the hyperparameters that yielded the highest cross-validated accuracy.\nIn the second article, we looked at searching for the best hyperparameters more conveniently using\nGridSearchCV\nin\nscikit-learn\n. If you remember, towards the end of the article, we touched upon the fact that performing Grid Search to tune hyperparameters is actually computationally inefficient.\nWe shall now try to rephrase it better, in a more formal way.\nSay we have to search for\nM\nparameters; Let\np_1, p_2,p_3, …, p_M\nbe the\nM\nparameters. Let the number of values that we would like to search over for\np_1\nbe\nn1\n, for\np_2\nbe\nn2\n, and so on, with\nnM\nvalues for\np_M\n.\nGrid Search considers all possible hyperparameter settings (combinations) into account and creates a model for each possible setting to choose the best model with optimal hyperparameters.\nTo understand it better, assume that out of M parameters, we decide to freeze the values of all hyperparameters except one, say the M_th parameter\np_M\n. So, Grid Search involves searching through the list of nM values for the M_th hyperparameter; So, there are\nnM\nmodels created.\nSuppose we now freeze the values of all hyperparameters except two, say the last two\n(p_M and p_(M-1))\n. We now have to search through\nall possible combinations\nof\np_M\nand\np_(M-1)\n, each having\nnM\nand\nn_(M-1)\npossible values that we could search over.\nWe now take a step back and freeze the value of\np_M-1\nand search through all values for\np_M\n; To account for all possible combinations, we should repeat the procedure for all\nn_M-1\nvalues for\np_M-1\n. So, this process would leave us with\nn_(M-1) * nM\nmodels.\nHope it’s clear how the complexity scales with increasing values of the number of values each hyperparameter could take. For the above example with M hyperparameters, we would have\nn1*n2*n3*…*n_M\nmodels.This is why we said that things could scale up quickly and become computationally intractable with Grid Search.\nWith this motivation to make hyperparameter search computationally more efficient, let us proceed to understand Randomized Search.\nUnderstanding RandomizedSearchCV\nIn contrast to\nGridSearchCV\n, not all parameter values are tried out in\nRandomizedSearchCV\n, but rather a fixed number of parameter settings is sampled from the specified distributions/ list of parameters.\nIf some of the hyperparameters that we’re searching for are continuous, then we should specify the distribution rather than the list of values, while defining the parameter grid. How do we define the fixed number of parameter settings?\nThe number of parameter settings that are tried is given by\nn_iter\n. There’s quality vs computational cost trade-off in picking\nn_iter\n.\nA very small value of\nn_iter\nwould imply that we’re more likely to find a suboptimal solution, because we are actually considering too few combinations.\nA very high value of\nn_iter\nwould mean we can ideally get closer to finding the best hyperparameters that yield the best model, but this again comes with a high computation cost as before. In fact, if we set\nn_iter= n1*n2*n3*…*n_M\nfrom the previous example, then, we’re essentially considering all possible hyperparameter combinations and now Randomized Search and Grid Search are equivalent.\nLet us build on the same example of\nKNNClassifier\nfrom the previous blog posts; In addition to\nn_neighbors\n, let us also search for the\noptimal weighting strategy\n— ‘\nuniform\n’ where all points are weighted equally and ‘\ndistance\n’ option weights points by the inverse of their distance. And now, let us implement Randomized Search in\nscikit-learn\nand do the following steps, as we did for Grid Search.\n1. Import RandomizedSearchCV class\nfrom\nsklearn.model_selection\nimport\nRandomizedSearchCV\nEnter fullscreen mode\nExit fullscreen mode\n2. Define the parameter grid\n# specify \"parameter distributions\" rather than a \"parameter grid\"\nparam_dist\n=\ndict\n(\nn_neighbors\n=\nk_range\n,\nweights\n=\nweight_options\n)\nEnter fullscreen mode\nExit fullscreen mode\n3. Instantiate the grid; Set n_iter=10, Fit the grid & View the results\n# n_iter controls the number of searches\nrand\n=\nRandomizedSearchCV\n(\nknn\n,\nparam_dist\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nn_iter\n=\n10\n,\nrandom_state\n=\n5\n,\nreturn_train_score\n=\nFalse\n)\nrand\n.\nfit\n(\nX\n,\ny\n)\npd\n.\nDataFrame\n(\nrand\n.\ncv_results_\n)[[\n'\nmean_test_score\n'\n,\n'\nstd_test_score\n'\n,\n'\nparams\n'\n]]\n#DataFrame\nmean_test_score\nstd_test_score\nparams\n0\n0.973333\n0.032660\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n16\n}\n1\n0.966667\n0.033333\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n22\n}\n2\n0.980000\n0.030551\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n18\n}\n3\n0.966667\n0.044721\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n27\n}\n4\n0.953333\n0.042687\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n29\n}\n5\n0.973333\n0.032660\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n10\n}\n6\n0.966667\n0.044721\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n22\n}\n7\n0.973333\n0.044222\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n14\n}\n8\n0.973333\n0.044222\n{\n'\nweights\n'\n:\n'\ndistance\n'\n,\n'\nn_neighbors\n'\n:\n12\n}\n9\n0.973333\n0.032660\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n15\n}\nEnter fullscreen mode\nExit fullscreen mode\n4. Examine the best score and best hyperparameters\n# examine the best model\nprint\n(\nrand\n.\nbest_score_\n)\nprint\n(\nrand\n.\nbest_params_\n)\n# Output\n0.9800000000000001\n{\n'\nweights\n'\n:\n'\nuniform\n'\n,\n'\nn_neighbors\n'\n:\n18\n}\nEnter fullscreen mode\nExit fullscreen mode\nParameters of the best model\nSurprisingly, we see that the highest accuracy score obtained in this case, where we only looked at 10 different parameter settings instead of 60 in Grid Search, is the same as before: 0.98 ✔\nAnd the value for n_neighbors= 18, which is also one of the optimal values that we got when we initially searched for the optimal value of n_neighbors. (Recall from the earlier blog post with the same example)\nMaybe we just got lucky?\nWhat is the guarantee that we will always get the best results?\nAh, this question makes perfect sense, doesn’t it?\nLet us do the following now: Let us run\nRandomizedSearchCV\nfor multiple times and see how many times we really end up getting lucky!\n➡️Run RandomizedSearchCV 20 times and see what happens; We log the best_score_ for every run.\n# run RandomizedSearchCV 20 times (with n_iter=10) and record the best score\nbest_scores\n=\n[]\nfor\n_\nin\nrange\n(\n20\n):\nrand\n=\nRandomizedSearchCV\n(\nknn\n,\nparam_dist\n,\ncv\n=\n10\n,\nscoring\n=\n'\naccuracy\n'\n,\nn_iter\n=\n10\n,\nreturn_train_score\n=\nFalse\n)\nrand\n.\nfit\n(\nX\n,\ny\n)\nbest_scores\n.\nappend\n(\nround\n(\nrand\n.\nbest_score_\n,\n3\n))\nLet\nus\nexamine\nall\nthe\n20\nbest\nscores\nnow\n.\nprint\n(\nbest_scores\n)\n# Output: Best Scores\n[\n0.973\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.973\n,\n0.98\n,\n0.98\n,\n0.973\n,\n0.98\n,\n0.973\n,\n0.973\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.973\n,\n0.98\n,\n0.98\n,\n0.98\n,\n0.973\n]\nEnter fullscreen mode\nExit fullscreen mode\nUpon examining the best scores above for all the 20 runs, we see that we get the best accuracy score of 0.98 about 13 times.\nLooks like we’re lucky indeed! What about the other 7 times when we didn't quite get the best accuracy score? These accuracy scores are around 0.973 which is pretty close to 0.98.\nThis observation convinces us that even though Randomized Search may not always give the hyperparameters of the best performing model, the models obtained by using these hyperparameters do not perform much worse compared to the best model obtained from Grid Search. This means, the best models thus obtained, with the hyperparameters from randomized search are clearly very close to the optimal model.\nIn essence, these may not be the best hyperparameters, but certainly close to the best hyperparameters, except that these are found under resource-constrained settings. Hope you all understood how we could use Randomized Search for hyperparameter tuning.\nThanks for reading! Happy Learning! Until next time!\nReferences\n[1] Here’s the\nlink\nto the Google Colab notebook for the example discussed above\n[2]\nIntroduction to Machine Learning in Python with scikit-learn by DataSchool\n.\n[3] Scikit-learn documentation- RandomizedSearchCV\nhttp://scikitlearn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html\nPhoto by\nChris Lawton\non\nUnsplash",
  "metadata": {
    "title": "Hyperparameter Tuning: Understanding Randomized Search - DEV Community",
    "description": "Hi everyone! This is the third of the series of articles on cross-validation and hyperparameter tunin... Tagged with python, machinelearning, beginners.",
    "keywords": "python, machinelearning, beginners, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/hyperparameter-tuning-understanding-randomized-search-343l",
    "og:title": "Hyperparameter Tuning: Understanding Randomized Search",
    "og:description": "Hi everyone! This is the third of the series of articles on cross-validation and hyperparameter tunin...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fluyul8idnjn0k9my64bu.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Hyperparameter Tuning: Understanding Randomized Search",
    "twitter:description": "Hi everyone! This is the third of the series of articles on cross-validation and hyperparameter tunin...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fluyul8idnjn0k9my64bu.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/28bc1d271498/code/body-000.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/28bc1d271498/code/body-001.txt",
      "chars": 130,
      "language": "text"
    },
    {
      "file": "attachments/28bc1d271498/code/body-002.txt",
      "chars": 1015,
      "language": "text"
    },
    {
      "file": "attachments/28bc1d271498/code/body-003.txt",
      "chars": 145,
      "language": "text"
    },
    {
      "file": "attachments/28bc1d271498/code/body-004.txt",
      "chars": 495,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://dev.to/balapriya/role-of-cross-validation-in-model-validation-and-hyperparameter-search-n3",
      "label": "Role of Cross-Validation in Model Validation and Hyperparameter Search",
      "type": "a"
    },
    {
      "href": "https://dev.to/balapriya/hyperparameter-tuning-understanding-grid-search-2648",
      "label": "Hyperparameter Tuning: Understanding Grid Search",
      "type": "a"
    },
    {
      "href": "https://colab.research.google.com/drive/1CRqP0a-L6zgOQY-w3U0qS7e4WvWtfR0P?usp=sharing",
      "label": "link",
      "type": "a"
    },
    {
      "href": "https://www.dataschool.io/machine-learning-with-scikit-learn/",
      "label": "Introduction to Machine Learning in Python with scikit-learn by DataSchool",
      "type": "a"
    },
    {
      "href": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html",
      "label": "http://scikitlearn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@chrislawton?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Chris Lawton",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/s/photos/random?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:12Z",
  "source_url": "https://dev.to/balapriya/hyperparameter-tuning-understanding-randomized-search-343l",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 8990,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/28bc1d271498/pages/source.html",
  "local_html": "attachments/28bc1d271498/pages/article.html",
  "body_base_path": "attachments/28bc1d271498/pages/",
  "body_markdown_path": "attachments/28bc1d271498/pages/article.md"
}