{
  "source_file": "0930",
  "source_platform": "技术平台",
  "author": "Jason Brownlee / Machine Learning Mastery",
  "title_listed": "How to Develop a Random Forest Model",
  "keywords": "随机森林,集成,机器学习",
  "domain": "机器学习",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://machinelearningmastery.com/random-forest-ensemble-in-python/",
  "publish_date": "2020-11",
  "corpus_stage": "预训练/SFT",
  "notes": "随机森林模型教程",
  "id": "9152a1982f64",
  "collected_at": "2026-10-02T17:59:48",
  "title": "How to Develop a Random Forest Ensemble in Python - MachineLearningMastery.com",
  "body_markdown": "How to Develop a Random Forest Ensemble in Python\nBy\nJason Brownlee\non\nApril 27, 2021\nin\nEnsemble Learning\n49\nShare\nPost\nShare\nRandom forest is an ensemble machine learning algorithm.\nIt is perhaps the most popular and widely used machine learning algorithm given its good or excellent performance across a wide range of classification and regression predictive modeling problems.\nIt is also easy to use given that it has few key hyperparameters and sensible heuristics for configuring these hyperparameters.\nIn this tutorial, you will discover how to develop a random forest ensemble for classification and regression.\nAfter completing this tutorial, you will know:\nRandom forest ensemble is an ensemble of decision trees and a natural extension of bagging.\nHow to use the random forest ensemble for classification and regression with scikit-learn.\nHow to explore the effect of random forest model hyperparameters on model performance.\nKick-start your project\nwith my new book\nEnsemble Learning Algorithms With Python\n, including\nstep-by-step tutorials\nand the\nPython source code\nfiles for all examples.\nLet’s get started.\nUpdate Aug/2020\n: Added a common questions section.\nHow to Develop a Random Forest Ensemble in Python\nPhoto by\nSheila Sund\n, some rights reserved.\nTutorial Overview\nThis tutorial is divided into four parts; they are:\nRandom Forest Algorithm\nRandom Forest Scikit-Learn API\nRandom Forest for Classification\nRandom Forest for Regression\nRandom Forest Hyperparameters\nExplore Number of Samples\nExplore Number of Features\nExplore Number of Trees\nExplore Tree Depth\nCommon Questions\nRandom Forest Algorithm\nRandom forest is an ensemble of decision tree algorithms.\nIt is an extension of\nbootstrap aggregation (bagging)\nof decision trees and can be used for classification and regression problems.\nIn bagging, a number of decision trees are created where each tree is created from a different bootstrap sample of the training dataset. A\nbootstrap sample\nis a sample of the training dataset where a sample may appear more than once in the sample, referred to as\nsampling with replacement\n.\nBagging is an effective ensemble algorithm as each decision tree is fit on a slightly different training dataset, and in turn, has a slightly different performance. Unlike normal decision tree models, such as\nclassification and regression trees\n(CART), trees used in the ensemble are unpruned, making them slightly overfit to the training dataset. This is desirable as it helps to make each tree more different and have less correlated predictions or prediction errors.\nPredictions from the trees are averaged across all decision trees resulting in better performance than any single tree in the model.\nEach model in the ensemble is then used to generate a prediction for a new sample and these m predictions are averaged to give the forest’s prediction\n— Page 199,\nApplied Predictive Modeling\n, 2013.\nA prediction on a regression problem is the average of the prediction across the trees in the ensemble. A prediction on a classification problem is the majority vote for the class label across the trees in the ensemble.\nRegression\n: Prediction is the average prediction across the decision trees.\nClassification\n: Prediction is the majority vote class label predicted across the decision trees.\nAs with bagging, each tree in the forest casts a vote for the classification of a new sample, and the proportion of votes in each class across the ensemble is the predicted probability vector.\n— Page 387,\nApplied Predictive Modeling\n, 2013.\nRandom forest involves constructing a large number of decision trees from bootstrap samples from the training dataset, like bagging.\nUnlike bagging, random forest also involves selecting a subset of input features (columns or variables) at each split point in the construction of trees. Typically, constructing a decision tree involves evaluating the value for each input variable in the data in order to select a split point. By reducing the features to a random subset that may be considered at each split point, it forces each decision tree in the ensemble to be more different.\nRandom forests provide an improvement over bagged trees by way of a small tweak that decorrelates the trees. […] But when building these decision trees, each time a split in a tree is considered, a random sample of m predictors is chosen as split candidates from the full set of p predictors.\n— Page 320,\nAn Introduction to Statistical Learning with Applications in R\n, 2014.\nThe effect is that the predictions, and in turn, prediction errors, made by each tree in the ensemble are more different or less correlated. When the predictions from these less correlated trees are averaged to make a prediction, it often results in better performance than bagged decision trees.\nPerhaps the most important hyperparameter to tune for the random forest is the number of random features to consider at each split point.\nRandom forests’ tuning parameter is the number of randomly selected predictors, k, to choose from at each split, and is commonly referred to as mtry. In the regression context, Breiman (2001) recommends setting mtry to be one-third of the number of predictors.\n— Page 199,\nApplied Predictive Modeling\n, 2013.\nA good heuristic for regression is to set this hyperparameter to 1/3 the number of input features.\nnum_features_for_split = total_input_features / 3\nFor classification problems, Breiman (2001) recommends setting mtry to the square root of the number of predictors.\n— Page 387,\nApplied Predictive Modeling\n, 2013.\nA good heuristic for classification is to set this hyperparameter to the square root of the number of input features.\nnum_features_for_split = sqrt(total_input_features)\nAnother important hyperparameter to tune is the depth of the decision trees. Deeper trees are often more overfit to the training data, but also less correlated, which in turn may improve the performance of the ensemble. Depths from 1 to 10 levels may be effective.\nFinally, the number of decision trees in the ensemble can be set. Often, this is increased until no further improvement is seen.\nWant to Get Started With Ensemble Learning?\nTake my free 7-day email crash course now (with sample code).\nClick to sign-up and also get a free PDF Ebook version of the course.\nDownload Your FREE Mini-Course\nRandom Forest Scikit-Learn API\nRandom Forest ensembles can be implemented from scratch, although this can be challenging for beginners.\nThe scikit-learn Python machine learning library provides an implementation of Random Forest for machine learning.\nIt is available in modern versions of the library.\nFirst, confirm that you are using a modern version of the library by running the following script:\n# check scikit-learn version\nimport sklearn\nprint(sklearn.__version__)\n1\n2\n3\n# check scikit-learn version\nimport\nsklearn\nprint\n(\nsklearn\n.\n__version__\n)\nRunning the script will print your version of scikit-learn.\nYour version should be the same or higher. If not, you must upgrade your version of the scikit-learn library.\n0.22.1\n1\n0.22.1\nRandom Forest is provided via the\nRandomForestRegressor\nand\nRandomForestClassifier\nclasses.\nBoth models operate the same way and take the same arguments that influence how the decision trees are created.\nRandomness is used in the construction of the model. This means that each time the algorithm is run on the same data, it will produce a slightly different model.\nWhen using machine learning algorithms that have a stochastic learning algorithm, it is good practice to evaluate them by averaging their performance across multiple runs or repeats of\ncross-validation\n. When fitting a final model, it may be desirable to either increase the number of trees until the variance of the model is reduced across repeated evaluations, or to fit multiple final models and average their predictions.\nLet’s take a look at how to develop a Random Forest ensemble for both classification and regression tasks.\nRandom Forest for Classification\nIn this section, we will look at using Random Forest for a classification problem.\nFirst, we can use the\nmake_classification() function\nto create a synthetic binary classification problem with 1,000 examples and 20 input features.\nThe complete example is listed below.\n# test classification dataset\nfrom sklearn.datasets import make_classification\n# define dataset\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)\n# summarize the dataset\nprint(X.shape, y.shape)\n1\n2\n3\n4\n5\n6\n# test classification dataset\nfrom\nsklearn\n.\ndatasets\nimport\nmake\n_\nclassification\n# define dataset\nX\n,\ny\n=\nmake_classification\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nn_redundant\n=\n5\n,\nrandom_state\n=\n3\n)\n# summarize the dataset\nprint\n(\nX\n.\nshape\n,\ny\n.\nshape\n)\nRunning the example creates the dataset and summarizes the shape of the input and output components.\n(1000, 20) (1000,)\n1\n(1000, 20) (1000,)\nNext, we can evaluate a random forest algorithm on this dataset.\nWe will evaluate the model using\nrepeated stratified k-fold cross-validation\n, with three repeats and 10 folds. We will report the mean and standard deviation of the accuracy of the model across all repeats and folds.\n# evaluate random forest algorithm for classification\nfrom numpy import mean\nfrom numpy import std\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import RepeatedStratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\n# define dataset\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)\n# define the model\nmodel = RandomForestClassifier()\n# evaluate the model\ncv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\nn_scores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1, error_score='raise')\n# report performance\nprint('Accuracy: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n# evaluate random forest algorithm for classification\nfrom\nnumpy\nimport\nmean\nfrom\nnumpy\nimport\nstd\nfrom\nsklearn\n.\ndatasets\nimport\nmake_classification\nfrom\nsklearn\n.\nmodel_selection\nimport\ncross_val_score\nfrom\nsklearn\n.\nmodel_selection\nimport\nRepeatedStratifiedKFold\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestClassifier\n# define dataset\nX\n,\ny\n=\nmake_classification\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nn_redundant\n=\n5\n,\nrandom_state\n=\n3\n)\n# define the model\nmodel\n=\nRandomForestClassifier\n(\n)\n# evaluate the model\ncv\n=\nRepeatedStratifiedKFold\n(\nn_splits\n=\n10\n,\nn_repeats\n=\n3\n,\nrandom_state\n=\n1\n)\nn_scores\n=\ncross_val_score\n(\nmodel\n,\nX\n,\ny\n,\nscoring\n=\n'accuracy'\n,\ncv\n=\ncv\n,\nn_jobs\n=\n-\n1\n,\nerror_score\n=\n'raise'\n)\n# report performance\nprint\n(\n'Accuracy: %.3f (%.3f)'\n%\n(\nmean\n(\nn_scores\n)\n,\nstd\n(\nn_scores\n)\n)\n)\nRunning the example reports the mean and standard deviation accuracy of the model.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see the random forest ensemble with default hyperparameters achieves a classification accuracy of about 90.5 percent.\nAccuracy: 0.905 (0.025)\n1\nAccuracy: 0.905 (0.025)\nWe can also use the random forest model as a final model and make predictions for classification.\nFirst, the random forest ensemble is fit on all available data, then the\npredict()\nfunction can be called to make predictions on new data.\nThe example below demonstrates this on our binary classification dataset.\n# make predictions using random forest for classification\nfrom sklearn.datasets import make_classification\nfrom sklearn.ensemble import RandomForestClassifier\n# define dataset\nX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)\n# define the model\nmodel = RandomForestClassifier()\n# fit the model on the whole dataset\nmodel.fit(X, y)\n# make a single prediction\nrow = [[-8.52381793,5.24451077,-12.14967704,-2.92949242,0.99314133,0.67326595,-0.38657932,1.27955683,-0.60712621,3.20807316,0.60504151,-1.38706415,8.92444588,-7.43027595,-2.33653219,1.10358169,0.21547782,1.05057966,0.6975331,0.26076035]]\nyhat = model.predict(row)\nprint('Predicted Class: %d' % yhat[0])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n# make predictions using random forest for classification\nfrom\nsklearn\n.\ndatasets\nimport\nmake_classification\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestClassifier\n# define dataset\nX\n,\ny\n=\nmake_classification\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nn_redundant\n=\n5\n,\nrandom_state\n=\n3\n)\n# define the model\nmodel\n=\nRandomForestClassifier\n(\n)\n# fit the model on the whole dataset\nmodel\n.\nfit\n(\nX\n,\ny\n)\n# make a single prediction\nrow\n=\n[\n[\n-\n8.52381793\n,\n5.24451077\n,\n-\n12.14967704\n,\n-\n2.92949242\n,\n0.99314133\n,\n0.67326595\n,\n-\n0.38657932\n,\n1.27955683\n,\n-\n0.60712621\n,\n3.20807316\n,\n0.60504151\n,\n-\n1.38706415\n,\n8.92444588\n,\n-\n7.43027595\n,\n-\n2.33653219\n,\n1.10358169\n,\n0.21547782\n,\n1.05057966\n,\n0.6975331\n,\n0.26076035\n]\n]\nyhat\n=\nmodel\n.\npredict\n(\nrow\n)\nprint\n(\n'Predicted Class: %d'\n%\nyhat\n[\n0\n]\n)\nRunning the example fits the random forest ensemble model on the entire dataset and is then used to make a prediction on a new row of data, as we might when using the model in an application.\nPredicted Class: 0\n1\nPredicted Class: 0\nNow that we are familiar with using random forest for classification, let’s look at the API for regression.\nRandom Forest for Regression\nIn this section, we will look at using random forests for a regression problem.\nFirst, we can use the\nmake_regression() function\nto create a synthetic regression problem with 1,000 examples and 20 input features.\nThe complete example is listed below.\n# test regression dataset\nfrom sklearn.datasets import make_regression\n# define dataset\nX, y = make_regression(n_samples=1000, n_features=20, n_informative=15, noise=0.1, random_state=2)\n# summarize the dataset\nprint(X.shape, y.shape)\n1\n2\n3\n4\n5\n6\n# test regression dataset\nfrom\nsklearn\n.\ndatasets\nimport\nmake\n_\nregression\n# define dataset\nX\n,\ny\n=\nmake_regression\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nnoise\n=\n0.1\n,\nrandom_state\n=\n2\n)\n# summarize the dataset\nprint\n(\nX\n.\nshape\n,\ny\n.\nshape\n)\nRunning the example creates the dataset and summarizes the shape of the input and output components.\n(1000, 20) (1000,)\n1\n(1000, 20) (1000,)\nNext, we can evaluate a random forest algorithm on this dataset.\nAs we did with the last section, we will evaluate the model using repeated k-fold cross-validation, with three repeats and 10 folds. We will report the mean absolute error (MAE) of the model across all repeats and folds. The scikit-learn library makes the MAE negative so that it is maximized instead of minimized. This means that larger negative MAE are better and a perfect model has a MAE of 0.\nThe complete example is listed below.\n# evaluate random forest ensemble for regression\nfrom numpy import mean\nfrom numpy import std\nfrom sklearn.datasets import make_regression\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import RepeatedKFold\nfrom sklearn.ensemble import RandomForestRegressor\n# define dataset\nX, y = make_regression(n_samples=1000, n_features=20, n_informative=15, noise=0.1, random_state=2)\n# define the model\nmodel = RandomForestRegressor()\n# evaluate the model\ncv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)\nn_scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1, error_score='raise')\n# report performance\nprint('MAE: %.3f (%.3f)' % (mean(n_scores), std(n_scores)))\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n# evaluate random forest ensemble for regression\nfrom\nnumpy\nimport\nmean\nfrom\nnumpy\nimport\nstd\nfrom\nsklearn\n.\ndatasets\nimport\nmake_regression\nfrom\nsklearn\n.\nmodel_selection\nimport\ncross_val_score\nfrom\nsklearn\n.\nmodel_selection\nimport\nRepeatedKFold\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestRegressor\n# define dataset\nX\n,\ny\n=\nmake_regression\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nnoise\n=\n0.1\n,\nrandom_state\n=\n2\n)\n# define the model\nmodel\n=\nRandomForestRegressor\n(\n)\n# evaluate the model\ncv\n=\nRepeatedKFold\n(\nn_splits\n=\n10\n,\nn_repeats\n=\n3\n,\nrandom_state\n=\n1\n)\nn_scores\n=\ncross_val_score\n(\nmodel\n,\nX\n,\ny\n,\nscoring\n=\n'neg_mean_absolute_error'\n,\ncv\n=\ncv\n,\nn_jobs\n=\n-\n1\n,\nerror_score\n=\n'raise'\n)\n# report performance\nprint\n(\n'MAE: %.3f (%.3f)'\n%\n(\nmean\n(\nn_scores\n)\n,\nstd\n(\nn_scores\n)\n)\n)\nRunning the example reports the mean and standard deviation MAE of the model.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see the random forest ensemble with default hyperparameters achieves a MAE of about 90.\nMAE: -90.149 (7.924)\n1\nMAE: -90.149 (7.924)\nWe can also use the random forest model as a final model and make predictions for regression.\nFirst, the random forest ensemble is fit on all available data, then the\npredict()\nfunction can be called to make predictions on new data.\nThe example below demonstrates this on our regression dataset.\n# random forest for making predictions for regression\nfrom sklearn.datasets import make_regression\nfrom sklearn.ensemble import RandomForestRegressor\n# define dataset\nX, y = make_regression(n_samples=1000, n_features=20, n_informative=15, noise=0.1, random_state=2)\n# define the model\nmodel = RandomForestRegressor()\n# fit the model on the whole dataset\nmodel.fit(X, y)\n# make a single prediction\nrow = [[-0.89483109,-1.0670149,-0.25448694,-0.53850126,0.21082105,1.37435592,0.71203659,0.73093031,-1.25878104,-2.01656886,0.51906798,0.62767387,0.96250155,1.31410617,-1.25527295,-0.85079036,0.24129757,-0.17571721,-1.11454339,0.36268268]]\nyhat = model.predict(row)\nprint('Prediction: %d' % yhat[0])\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n# random forest for making predictions for regression\nfrom\nsklearn\n.\ndatasets\nimport\nmake_regression\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestRegressor\n# define dataset\nX\n,\ny\n=\nmake_regression\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nnoise\n=\n0.1\n,\nrandom_state\n=\n2\n)\n# define the model\nmodel\n=\nRandomForestRegressor\n(\n)\n# fit the model on the whole dataset\nmodel\n.\nfit\n(\nX\n,\ny\n)\n# make a single prediction\nrow\n=\n[\n[\n-\n0.89483109\n,\n-\n1.0670149\n,\n-\n0.25448694\n,\n-\n0.53850126\n,\n0.21082105\n,\n1.37435592\n,\n0.71203659\n,\n0.73093031\n,\n-\n1.25878104\n,\n-\n2.01656886\n,\n0.51906798\n,\n0.62767387\n,\n0.96250155\n,\n1.31410617\n,\n-\n1.25527295\n,\n-\n0.85079036\n,\n0.24129757\n,\n-\n0.17571721\n,\n-\n1.11454339\n,\n0.36268268\n]\n]\nyhat\n=\nmodel\n.\npredict\n(\nrow\n)\nprint\n(\n'Prediction: %d'\n%\nyhat\n[\n0\n]\n)\nRunning the example fits the random forest ensemble model on the entire dataset and is then used to make a prediction on a new row of data, as we might when using the model in an application.\nPrediction: -173\n1\nPrediction: -173\nNow that we are familiar with using the scikit-learn API to evaluate and use random forest ensembles, let’s look at configuring the model.\nRandom Forest Hyperparameters\nIn this section, we will take a closer look at some of the hyperparameters you should consider tuning for the random forest ensemble and their effect on model performance.\nExplore Number of Samples\nEach decision tree in the ensemble is fit on a\nbootstrap sample\ndrawn from the training dataset.\nThis can be turned off by setting the “\nbootstrap\n” argument to\nFalse\n, if you desire. In that case, the whole training dataset will be used to train each decision tree.\nThis is not recommended\n.\nThe “\nmax_samples\n” argument can be set to a float between 0 and 1 to control the percentage of the size of the training dataset to make the bootstrap sample used to train each decision tree.\nFor example, if the training dataset has 100 rows, the\nmax_samples\nargument could be set to 0.5 and each decision tree will be fit on a bootstrap sample with (100 * 0.5) or 50 rows of data.\nA smaller sample size will make trees more different, and a larger sample size will make the trees more similar. Setting\nmax_samples\nto “\nNone\n” will make the sample size the same size as the training dataset and this is the default.\nThe example below demonstrates the effect of different bootstrap sample sizes from 10 percent to 100 percent on the random forest algorithm.\n# explore random forest bootstrap sample size on performance\nfrom numpy import mean\nfrom numpy import std\nfrom numpy import arange\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import RepeatedStratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\nfrom matplotlib import pyplot\n\n# get the dataset\ndef get_dataset():\n\tX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)\n\treturn X, y\n\n# get a list of models to evaluate\ndef get_models():\n\tmodels = dict()\n\t# explore ratios from 10% to 100% in 10% increments\n\tfor i in arange(0.1, 1.1, 0.1):\n\t\tkey = '%.1f' % i\n\t\t# set max_samples=None to use 100%\n\t\tif i == 1.0:\n\t\t\ti = None\n\t\tmodels[key] = RandomForestClassifier(max_samples=i)\n\treturn models\n\n# evaluate a given model using cross-validation\ndef evaluate_model(model, X, y):\n\t# define the evaluation procedure\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n\t# evaluate the model and collect the results\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n\treturn scores\n\n# define dataset\nX, y = get_dataset()\n# get the models to evaluate\nmodels = get_models()\n# evaluate the models and store results\nresults, names = list(), list()\nfor name, model in models.items():\n\t# evaluate the model\n\tscores = evaluate_model(model, X, y)\n\t# store the results\n\tresults.append(scores)\n\tnames.append(name)\n\t# summarize the performance along the way\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n# plot model performance for comparison\npyplot.boxplot(results, labels=names, showmeans=True)\npyplot.show()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n49\n50\n51\n52\n# explore random forest bootstrap sample size on performance\nfrom\nnumpy\nimport\nmean\nfrom\nnumpy\nimport\nstd\nfrom\nnumpy\nimport\narange\nfrom\nsklearn\n.\ndatasets\nimport\nmake_classification\nfrom\nsklearn\n.\nmodel_selection\nimport\ncross_val_score\nfrom\nsklearn\n.\nmodel_selection\nimport\nRepeatedStratifiedKFold\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestClassifier\nfrom\nmatplotlib\nimport\npyplot\n# get the dataset\ndef\nget_dataset\n(\n)\n:\nX\n,\ny\n=\nmake_classification\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nn_redundant\n=\n5\n,\nrandom_state\n=\n3\n)\nreturn\nX\n,\ny\n# get a list of models to evaluate\ndef\nget_models\n(\n)\n:\nmodels\n=\ndict\n(\n)\n# explore ratios from 10% to 100% in 10% increments\nfor\ni\nin\narange\n(\n0.1\n,\n1.1\n,\n0.1\n)\n:\nkey\n=\n'%.1f'\n%\ni\n# set max_samples=None to use 100%\nif\ni\n==\n1.0\n:\ni\n=\nNone\nmodels\n[\nkey\n]\n=\nRandomForestClassifier\n(\nmax_samples\n=\ni\n)\nreturn\nmodels\n# evaluate a given model using cross-validation\ndef\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n:\n# define the evaluation procedure\ncv\n=\nRepeatedStratifiedKFold\n(\nn_splits\n=\n10\n,\nn_repeats\n=\n3\n,\nrandom_state\n=\n1\n)\n# evaluate the model and collect the results\nscores\n=\ncross_val_score\n(\nmodel\n,\nX\n,\ny\n,\nscoring\n=\n'accuracy'\n,\ncv\n=\ncv\n,\nn_jobs\n=\n-\n1\n)\nreturn\nscores\n# define dataset\nX\n,\ny\n=\nget_dataset\n(\n)\n# get the models to evaluate\nmodels\n=\nget_models\n(\n)\n# evaluate the models and store results\nresults\n,\nnames\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\nname\n,\nmodel\nin\nmodels\n.\nitems\n(\n)\n:\n# evaluate the model\nscores\n=\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n# store the results\nresults\n.\nappend\n(\nscores\n)\nnames\n.\nappend\n(\nname\n)\n# summarize the performance along the way\nprint\n(\n'>%s %.3f (%.3f)'\n%\n(\nname\n,\nmean\n(\nscores\n)\n,\nstd\n(\nscores\n)\n)\n)\n# plot model performance for comparison\npyplot\n.\nboxplot\n(\nresults\n,\nlabels\n=\nnames\n,\nshowmeans\n=\nTrue\n)\npyplot\n.\nshow\n(\n)\nRunning the example first reports the mean accuracy for each dataset size.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, the results suggest that using a bootstrap sample size that is equal to the size of the training dataset achieves the best results on this dataset.\nThis is the default and it should probably be used in most cases.\n>10 0.856 (0.031)\n>20 0.873 (0.029)\n>30 0.881 (0.021)\n>40 0.891 (0.033)\n>50 0.893 (0.025)\n>60 0.897 (0.030)\n>70 0.902 (0.024)\n>80 0.903 (0.024)\n>90 0.900 (0.026)\n>100 0.903 (0.027)\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n>10 0.856 (0.031)\n>20 0.873 (0.029)\n>30 0.881 (0.021)\n>40 0.891 (0.033)\n>50 0.893 (0.025)\n>60 0.897 (0.030)\n>70 0.902 (0.024)\n>80 0.903 (0.024)\n>90 0.900 (0.026)\n>100 0.903 (0.027)\nA box and whisker plot is created for the distribution of accuracy scores for each bootstrap sample size.\nIn this case, we can see a general trend that the larger the sample, the better the performance of the model.\nYou might like to extend this example and see what happens if the bootstrap sample size is larger or even much larger than the training dataset (e.g. you can set an integer value as the number of samples instead of a float percentage of the training dataset size).\nBox Plot of Random Forest Bootstrap Sample Size vs. Classification Accuracy\nExplore Number of Features\nThe number of features that is randomly sampled for each split point is perhaps the most important feature to configure for random forest.\nIt is set via the\nmax_features\nargument and defaults to the square root of the number of input features. In this case, for our test dataset, this would be\nsqrt(20)\nor about four features.\nThe example below explores the effect of the number of features randomly selected at each split point on model accuracy. We will try values from 1 to 7 and would expect a small value, around four, to perform well based on the heuristic.\n# explore random forest number of features effect on performance\nfrom numpy import mean\nfrom numpy import std\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import RepeatedStratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\nfrom matplotlib import pyplot\n\n# get the dataset\ndef get_dataset():\n\tX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)\n\treturn X, y\n\n# get a list of models to evaluate\ndef get_models():\n\tmodels = dict()\n\t# explore number of features from 1 to 7\n\tfor i in range(1,8):\n\t\tmodels[str(i)] = RandomForestClassifier(max_features=i)\n\treturn models\n\n# evaluate a given model using cross-validation\ndef evaluate_model(model, X, y):\n\t# define the evaluation procedure\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n\t# evaluate the model and collect the results\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n\treturn scores\n\n# define dataset\nX, y = get_dataset()\n# get the models to evaluate\nmodels = get_models()\n# evaluate the models and store results\nresults, names = list(), list()\nfor name, model in models.items():\n\t# evaluate the model\n\tscores = evaluate_model(model, X, y)\n\t# store the results\n\tresults.append(scores)\n\tnames.append(name)\n\t# summarize the performance along the way\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n# plot model performance for comparison\npyplot.boxplot(results, labels=names, showmeans=True)\npyplot.show()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n# explore random forest number of features effect on performance\nfrom\nnumpy\nimport\nmean\nfrom\nnumpy\nimport\nstd\nfrom\nsklearn\n.\ndatasets\nimport\nmake_classification\nfrom\nsklearn\n.\nmodel_selection\nimport\ncross_val_score\nfrom\nsklearn\n.\nmodel_selection\nimport\nRepeatedStratifiedKFold\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestClassifier\nfrom\nmatplotlib\nimport\npyplot\n# get the dataset\ndef\nget_dataset\n(\n)\n:\nX\n,\ny\n=\nmake_classification\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nn_redundant\n=\n5\n,\nrandom_state\n=\n3\n)\nreturn\nX\n,\ny\n# get a list of models to evaluate\ndef\nget_models\n(\n)\n:\nmodels\n=\ndict\n(\n)\n# explore number of features from 1 to 7\nfor\ni\nin\nrange\n(\n1\n,\n8\n)\n:\nmodels\n[\nstr\n(\ni\n)\n]\n=\nRandomForestClassifier\n(\nmax_features\n=\ni\n)\nreturn\nmodels\n# evaluate a given model using cross-validation\ndef\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n:\n# define the evaluation procedure\ncv\n=\nRepeatedStratifiedKFold\n(\nn_splits\n=\n10\n,\nn_repeats\n=\n3\n,\nrandom_state\n=\n1\n)\n# evaluate the model and collect the results\nscores\n=\ncross_val_score\n(\nmodel\n,\nX\n,\ny\n,\nscoring\n=\n'accuracy'\n,\ncv\n=\ncv\n,\nn_jobs\n=\n-\n1\n)\nreturn\nscores\n# define dataset\nX\n,\ny\n=\nget_dataset\n(\n)\n# get the models to evaluate\nmodels\n=\nget_models\n(\n)\n# evaluate the models and store results\nresults\n,\nnames\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\nname\n,\nmodel\nin\nmodels\n.\nitems\n(\n)\n:\n# evaluate the model\nscores\n=\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n# store the results\nresults\n.\nappend\n(\nscores\n)\nnames\n.\nappend\n(\nname\n)\n# summarize the performance along the way\nprint\n(\n'>%s %.3f (%.3f)'\n%\n(\nname\n,\nmean\n(\nscores\n)\n,\nstd\n(\nscores\n)\n)\n)\n# plot model performance for comparison\npyplot\n.\nboxplot\n(\nresults\n,\nlabels\n=\nnames\n,\nshowmeans\n=\nTrue\n)\npyplot\n.\nshow\n(\n)\nRunning the example first reports the mean accuracy for each feature set size.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, the results suggest that a value between three and five would be appropriate, confirming the sensible default of four on this dataset. A value of five might even be better given the smaller standard deviation in classification accuracy as compared to a value of three or four.\n>1 0.897 (0.023)\n>2 0.900 (0.028)\n>3 0.903 (0.027)\n>4 0.903 (0.022)\n>5 0.903 (0.019)\n>6 0.898 (0.025)\n>7 0.900 (0.024)\n1\n2\n3\n4\n5\n6\n7\n>1 0.897 (0.023)\n>2 0.900 (0.028)\n>3 0.903 (0.027)\n>4 0.903 (0.022)\n>5 0.903 (0.019)\n>6 0.898 (0.025)\n>7 0.900 (0.024)\nA box and whisker plot is created for the distribution of accuracy scores for each feature set size.\nWe can see a trend in performance rising and peaking with values between three and five and falling again as larger feature set sizes are considered.\nBox Plot of Random Forest Feature Set Size vs. Classification Accuracy\nExplore Number of Trees\nThe number of trees is another key hyperparameter to configure for the random forest.\nTypically, the number of trees is increased until the model performance stabilizes. Intuition might suggest that more trees will lead to overfitting, although this is not the case. Both bagging and random forest algorithms appear to be somewhat immune to overfitting the training dataset given the stochastic nature of the learning algorithm.\nThe number of trees can be set via the “\nn_estimators\n” argument and defaults to 100.\nThe example below explores the effect of the number of trees with values between 10 to 1,000.\n# explore random forest number of trees effect on performance\nfrom numpy import mean\nfrom numpy import std\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import RepeatedStratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\nfrom matplotlib import pyplot\n\n# get the dataset\ndef get_dataset():\n\tX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)\n\treturn X, y\n\n# get a list of models to evaluate\ndef get_models():\n\tmodels = dict()\n\t# define number of trees to consider\n\tn_trees = [10, 50, 100, 500, 1000]\n\tfor n in n_trees:\n\t\tmodels[str(n)] = RandomForestClassifier(n_estimators=n)\n\treturn models\n\n# evaluate a given model using cross-validation\ndef evaluate_model(model, X, y):\n\t# define the evaluation procedure\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n\t# evaluate the model and collect the results\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n\treturn scores\n\n# define dataset\nX, y = get_dataset()\n# get the models to evaluate\nmodels = get_models()\n# evaluate the models and store results\nresults, names = list(), list()\nfor name, model in models.items():\n\t# evaluate the model\n\tscores = evaluate_model(model, X, y)\n\t# store the results\n\tresults.append(scores)\n\tnames.append(name)\n\t# summarize the performance along the way\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n# plot model performance for comparison\npyplot.boxplot(results, labels=names, showmeans=True)\npyplot.show()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n# explore random forest number of trees effect on performance\nfrom\nnumpy\nimport\nmean\nfrom\nnumpy\nimport\nstd\nfrom\nsklearn\n.\ndatasets\nimport\nmake_classification\nfrom\nsklearn\n.\nmodel_selection\nimport\ncross_val_score\nfrom\nsklearn\n.\nmodel_selection\nimport\nRepeatedStratifiedKFold\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestClassifier\nfrom\nmatplotlib\nimport\npyplot\n# get the dataset\ndef\nget_dataset\n(\n)\n:\nX\n,\ny\n=\nmake_classification\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nn_redundant\n=\n5\n,\nrandom_state\n=\n3\n)\nreturn\nX\n,\ny\n# get a list of models to evaluate\ndef\nget_models\n(\n)\n:\nmodels\n=\ndict\n(\n)\n# define number of trees to consider\nn_trees\n=\n[\n10\n,\n50\n,\n100\n,\n500\n,\n1000\n]\nfor\nn\nin\nn_trees\n:\nmodels\n[\nstr\n(\nn\n)\n]\n=\nRandomForestClassifier\n(\nn_estimators\n=\nn\n)\nreturn\nmodels\n# evaluate a given model using cross-validation\ndef\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n:\n# define the evaluation procedure\ncv\n=\nRepeatedStratifiedKFold\n(\nn_splits\n=\n10\n,\nn_repeats\n=\n3\n,\nrandom_state\n=\n1\n)\n# evaluate the model and collect the results\nscores\n=\ncross_val_score\n(\nmodel\n,\nX\n,\ny\n,\nscoring\n=\n'accuracy'\n,\ncv\n=\ncv\n,\nn_jobs\n=\n-\n1\n)\nreturn\nscores\n# define dataset\nX\n,\ny\n=\nget_dataset\n(\n)\n# get the models to evaluate\nmodels\n=\nget_models\n(\n)\n# evaluate the models and store results\nresults\n,\nnames\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\nname\n,\nmodel\nin\nmodels\n.\nitems\n(\n)\n:\n# evaluate the model\nscores\n=\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n# store the results\nresults\n.\nappend\n(\nscores\n)\nnames\n.\nappend\n(\nname\n)\n# summarize the performance along the way\nprint\n(\n'>%s %.3f (%.3f)'\n%\n(\nname\n,\nmean\n(\nscores\n)\n,\nstd\n(\nscores\n)\n)\n)\n# plot model performance for comparison\npyplot\n.\nboxplot\n(\nresults\n,\nlabels\n=\nnames\n,\nshowmeans\n=\nTrue\n)\npyplot\n.\nshow\n(\n)\nRunning the example first reports the mean accuracy for each configured number of trees.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see that performance rises and stays flat after about 100 trees. Mean accuracy scores fluctuate across 100, 500, and 1,000 trees and this may be statistical noise.\n>10 0.870 (0.036)\n>50 0.900 (0.028)\n>100 0.910 (0.024)\n>500 0.904 (0.024)\n>1000 0.906 (0.023)\n1\n2\n3\n4\n5\n>10 0.870 (0.036)\n>50 0.900 (0.028)\n>100 0.910 (0.024)\n>500 0.904 (0.024)\n>1000 0.906 (0.023)\nA box and whisker plot is created for the distribution of accuracy scores for each configured number of trees.\nBox Plot of Random Forest Ensemble Size vs. Classification Accuracy\nExplore Tree Depth\nA final interesting hyperparameter is the maximum depth of decision trees used in the ensemble.\nBy default, trees are constructed to an arbitrary depth and are not pruned. This is a sensible default, although we can also explore fitting trees with different fixed depths.\nThe maximum tree depth can be specified via the\nmax_depth\nargument and is set to\nNone\n(no maximum depth) by default.\nThe example below explores the effect of random forest maximum tree depth on model performance.\n# explore random forest tree depth effect on performance\nfrom numpy import mean\nfrom numpy import std\nfrom sklearn.datasets import make_classification\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import RepeatedStratifiedKFold\nfrom sklearn.ensemble import RandomForestClassifier\nfrom matplotlib import pyplot\n\n# get the dataset\ndef get_dataset():\n\tX, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_redundant=5, random_state=3)\n\treturn X, y\n\n# get a list of models to evaluate\ndef get_models():\n\tmodels = dict()\n\t# consider tree depths from 1 to 7 and None=full\n\tdepths = [i for i in range(1,8)] + [None]\n\tfor n in depths:\n\t\tmodels[str(n)] = RandomForestClassifier(max_depth=n)\n\treturn models\n\n# evaluate a given model using cross-validation\ndef evaluate_model(model, X, y):\n\t# define the evaluation procedure\n\tcv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)\n\t# evaluate the model and collect the results\n\tscores = cross_val_score(model, X, y, scoring='accuracy', cv=cv, n_jobs=-1)\n\treturn scores\n\n# define dataset\nX, y = get_dataset()\n# get the models to evaluate\nmodels = get_models()\n# evaluate the models and store results\nresults, names = list(), list()\nfor name, model in models.items():\n\t# evaluate the model\n\tscores = evaluate_model(model, X, y)\n\t# store the results\n\tresults.append(scores)\n\tnames.append(name)\n\t# summarize the performance along the way\n\tprint('>%s %.3f (%.3f)' % (name, mean(scores), std(scores)))\n# plot model performance for comparison\npyplot.boxplot(results, labels=names, showmeans=True)\npyplot.show()\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\n33\n34\n35\n36\n37\n38\n39\n40\n41\n42\n43\n44\n45\n46\n47\n48\n# explore random forest tree depth effect on performance\nfrom\nnumpy\nimport\nmean\nfrom\nnumpy\nimport\nstd\nfrom\nsklearn\n.\ndatasets\nimport\nmake_classification\nfrom\nsklearn\n.\nmodel_selection\nimport\ncross_val_score\nfrom\nsklearn\n.\nmodel_selection\nimport\nRepeatedStratifiedKFold\nfrom\nsklearn\n.\nensemble\nimport\nRandomForestClassifier\nfrom\nmatplotlib\nimport\npyplot\n# get the dataset\ndef\nget_dataset\n(\n)\n:\nX\n,\ny\n=\nmake_classification\n(\nn_samples\n=\n1000\n,\nn_features\n=\n20\n,\nn_informative\n=\n15\n,\nn_redundant\n=\n5\n,\nrandom_state\n=\n3\n)\nreturn\nX\n,\ny\n# get a list of models to evaluate\ndef\nget_models\n(\n)\n:\nmodels\n=\ndict\n(\n)\n# consider tree depths from 1 to 7 and None=full\ndepths\n=\n[\ni\nfor\ni\nin\nrange\n(\n1\n,\n8\n)\n]\n+\n[\nNone\n]\nfor\nn\nin\ndepths\n:\nmodels\n[\nstr\n(\nn\n)\n]\n=\nRandomForestClassifier\n(\nmax_depth\n=\nn\n)\nreturn\nmodels\n# evaluate a given model using cross-validation\ndef\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n:\n# define the evaluation procedure\ncv\n=\nRepeatedStratifiedKFold\n(\nn_splits\n=\n10\n,\nn_repeats\n=\n3\n,\nrandom_state\n=\n1\n)\n# evaluate the model and collect the results\nscores\n=\ncross_val_score\n(\nmodel\n,\nX\n,\ny\n,\nscoring\n=\n'accuracy'\n,\ncv\n=\ncv\n,\nn_jobs\n=\n-\n1\n)\nreturn\nscores\n# define dataset\nX\n,\ny\n=\nget_dataset\n(\n)\n# get the models to evaluate\nmodels\n=\nget_models\n(\n)\n# evaluate the models and store results\nresults\n,\nnames\n=\nlist\n(\n)\n,\nlist\n(\n)\nfor\nname\n,\nmodel\nin\nmodels\n.\nitems\n(\n)\n:\n# evaluate the model\nscores\n=\nevaluate_model\n(\nmodel\n,\nX\n,\ny\n)\n# store the results\nresults\n.\nappend\n(\nscores\n)\nnames\n.\nappend\n(\nname\n)\n# summarize the performance along the way\nprint\n(\n'>%s %.3f (%.3f)'\n%\n(\nname\n,\nmean\n(\nscores\n)\n,\nstd\n(\nscores\n)\n)\n)\n# plot model performance for comparison\npyplot\n.\nboxplot\n(\nresults\n,\nlabels\n=\nnames\n,\nshowmeans\n=\nTrue\n)\npyplot\n.\nshow\n(\n)\nRunning the example first reports the mean accuracy for each configured maximum tree depth.\nNote\n: Your\nresults may vary\ngiven the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.\nIn this case, we can see that larger depth results in better model performance, with the default of no maximum depth achieving the best performance on this dataset.\n>1 0.771 (0.040)\n>2 0.807 (0.037)\n>3 0.834 (0.034)\n>4 0.857 (0.030)\n>5 0.872 (0.025)\n>6 0.887 (0.024)\n>7 0.890 (0.025)\n>None 0.903 (0.027)\n1\n2\n3\n4\n5\n6\n7\n8\n>1 0.771 (0.040)\n>2 0.807 (0.037)\n>3 0.834 (0.034)\n>4 0.857 (0.030)\n>5 0.872 (0.025)\n>6 0.887 (0.024)\n>7 0.890 (0.025)\n>None 0.903 (0.027)\nA box and whisker plot is created for the distribution of accuracy scores for each configured maximum tree depth.\nIn this case, we can see a trend of improved performance with increase in tree depth, supporting the default of no maximum depth.\nBox Plot of Random Forest Maximum Tree Depth vs. Classification Accuracy\nCommon Questions\nIn this section we will take a closer look at some common sticking points you may have with the radom forest ensemble procedure.\nQ. What algorithm should be used in the ensemble?\nRandom forest is designed to be an ensemble of decision tree algorithms.\nQ. How many ensemble members should be used?\nThe number of trees should be increased until no further improvement in performance is seen on your dataset.\nAs a starting point, we suggest using at least 1,000 trees. If the cross-validation performance profiles are still improving at 1,000 trees, then incorporate more trees until performance levels off.\n— Page 200,\nApplied Predictive Modeling\n, 2013.\nQ. Won’t the ensemble overfit with too many trees?\nNo. Random forest ensembles (do not) are very unlikely to overfit in general.\nAnother claim is that random forests “cannot overfit” the data. It is certainly true that increasing [the number of trees] does not cause the random forest sequence to overfit …\n— Page 596,\nThe Elements of Statistical Learning\n, 2016.\nQ. How large should the bootstrap sample be?\nIt is good practice to make the bootstrap sample as large as the original dataset size.\nThat is 100% the size or an equal number of rows as the original dataset.\nQ. How many features should be chosen at each split point?\nThe best practice is to test a suite of different values and discover what works best for your dataset.\nAs a heuristic, you can use:\nClassification\n: Square root of the number of features.\nRegression\n: One third of the number of features.\nQ. What problems are well suited to random forest?\nRandom forest is known to work well or even best on a wide range of classification and regression problems. Try it and see.\nThe authors make grand claims about the success of random forests: “most accurate”, “most interpretable”, and the like. In our experience random forests do remarkably well, with very little tuning required.\n— Page 590,\nThe Elements of Statistical Learning\n, 2016.\nFurther Reading\nThis section provides more resources on the topic if you are looking to go deeper.\nTutorials\nHow to Implement Random Forest From Scratch in Python\nPapers\nRandom Forests\n, 2001.\nBooks\nApplied Predictive Modeling\n, 2013.\nThe Elements of Statistical Learning\n, 2016.\nAn Introduction to Statistical Learning with Applications in R\n, 2014.\nAPIs\nsklearn.ensemble.RandomForestRegressor API\n.\nsklearn.ensemble.RandomForestClassifier API\n.\nArticles\nRandom Forest, Wikipedia\n.\nSummary\nIn this tutorial, you discovered how to develop random forest ensembles for classification and regression.\nSpecifically, you learned:\nRandom forest ensemble is an ensemble of decision trees and a natural extension of bagging.\nHow to use the random forest ensemble for classification and regression with scikit-learn.\nHow to explore the effect of random forest model hyperparameters on model performance.\nDo you have any questions?\nAsk your questions in the comments below and I will do my best to answer.\nGet a Handle on Modern Ensemble Learning!\nImprove Your Predictions in Minutes\n...with just a few lines of python code\nDiscover how in my new Ebook:\nEnsemble Learning Algorithms With Python\nIt provides\nself-study tutorials\nwith\nfull working code\non:\nStacking\n,\nVoting\n,\nBoosting\n,\nBagging\n,\nBlending\n,\nSuper Learner\n, \n\tand much more...\nBring Modern Ensemble Learning Techniques to\nYour Machine Learning Projects\nSee What's Inside\nShare\nPost\nShare\nMore On This Topic\nHow to Develop a Random Subspace Ensemble With Python\nBagging and Random Forest Ensemble Algorithms for…\nHow to Develop Random Forest Ensembles With XGBoost\nHow to Develop a Weighted Average Ensemble With Python\nHow to Develop a Feature Selection Subspace Ensemble…\nHow to Develop a Gradient Boosting Machine Ensemble…\nAbout Jason Brownlee\nJason Brownlee, PhD is a machine learning specialist who teaches developers how to get results with modern machine learning methods via hands-on tutorials.\nView all posts by Jason Brownlee\n→",
  "metadata": {
    "title": "How to Develop a Random Forest Ensemble in Python - MachineLearningMastery.com",
    "description": null,
    "keywords": null,
    "author": "Jason Brownlee",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "How to Develop a Random Forest Ensemble in Python - MachineLearningMastery.com",
    "og:description": "Random forest is an ensemble machine learning algorithm. It is perhaps the most popular and widely used machine learning algorithm given its good or excellent performance across a wide range of classification and regression predictive modeling problems. It is also easy to use given that it has few key hyperparameters and sensible heuristics for configuring these hyperparameters. In this tutorial,…",
    "og:url": "https://machinelearningmastery.com/random-forest-ensemble-in-python/",
    "og:site_name": "MachineLearningMastery.com",
    "og:image": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Bootstrap-Sample-Size-vs-Classification-Accuracy.png",
    "og:image:width": "1280",
    "og:image:height": "960",
    "og:image:type": "image/png",
    "twitter:label1": "Written by",
    "twitter:data1": "Jason Brownlee",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "21 minutes",
    "twitter:card": "summary_large_image",
    "twitter:description": "Random forest is an ensemble machine learning algorithm. It is perhaps the most popular and widely used machine learning algorithm given its good or excellent performance across a wide range of classification and regression predictive modeling problems. It is also easy to use given that it has few key hyperparameters and sensible heuristics for configuring these hyperparameters. In this tutorial,…",
    "twitter:title": "How to Develop a Random Forest Ensemble in Python - MachineLearningMastery.com",
    "article:publisher": "https://www.facebook.com/MachineLearningMastery/",
    "article:author": "https://www.facebook.com/MachineLearningMastery/",
    "article:published_time": "2020-04-19T19:00:39+00:00",
    "article:modified_time": "2021-04-26T21:55:11+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/04/How-to-Develop-a-Random-Forest-Ensemble-in-Python.jpg",
      "bytes": 117586,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/000.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/04/How-to-Develop-a-Random-Forest-Ensemble-in-Python.jpg",
      "alt": "How to Develop a Random Forest Ensemble in Python",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Bootstrap-Sample-Size-vs-Classification-Accuracy.png",
      "bytes": 10458,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/003.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Bootstrap-Sample-Size-vs-Classification-Accuracy.png",
      "alt": "Box Plot of Random Forest Bootstrap Sample Size vs. Classification Accuracy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Feature-Set-Size-vs-Classification-Accuracy.png",
      "bytes": 6873,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/007.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Feature-Set-Size-vs-Classification-Accuracy.png",
      "alt": "Box Plot of Random Forest Feature Set Size vs. Classification Accuracy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Ensemble-Size-vs-Classification-Accuracy.png",
      "bytes": 7957,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/011.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Ensemble-Size-vs-Classification-Accuracy.png",
      "alt": "Box Plot of Random Forest Ensemble Size vs. Classification Accuracy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Maximum-Tree-Depth-vs-Classification-Accuracy.png",
      "bytes": 7377,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/015.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/04/Box-Plot-of-Random-Forest-Maximum-Tree-Depth-vs-Classification-Accuracy.png",
      "alt": "Box Plot of Random Forest Maximum Tree Depth vs. Classification Accuracy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2022/11/ELA4ML-220.png",
      "bytes": 5810,
      "reused": false,
      "local_path": "attachments/9152a1982f64/images/body-36aa300e87187946f3a5.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2022/11/ELA4ML-220.png",
      "alt": "Ensemble Learning Algorithms With Python",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/11/Box-Plot-of-Random-Subspace-Ensemble-Features-vs.-Classification-Accuracy.png",
      "bytes": 10375,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/020.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/11/Box-Plot-of-Random-Subspace-Ensemble-Features-vs.-Classification-Accuracy.png",
      "alt": "Box Plot of Random Subspace Ensemble Features vs. Classification Accuracy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2016/05/Bagging-and-Random-Forest-Ensemble-Algorithms-for-Machine-Learning.jpg",
      "bytes": 93135,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/024.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2016/05/Bagging-and-Random-Forest-Ensemble-Algorithms-for-Machine-Learning.jpg",
      "alt": "Bagging and Random Forest Ensemble Algorithms for Machine Learning",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/08/Box-Plots-of-XGBoost-Random-Forest-Feature-Set-Size-vs.-Classification-Accuracy.png",
      "bytes": 9813,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/026.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/08/Box-Plots-of-XGBoost-Random-Forest-Feature-Set-Size-vs.-Classification-Accuracy.png",
      "alt": "Box Plots of XGBoost Random Forest Feature Set Size vs. Classification Accuracy",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/12/How-to-Develop-a-Weighted-Average-Ensemble-With-Python.jpg",
      "bytes": 48384,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/030.jpg",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/12/How-to-Develop-a-Weighted-Average-Ensemble-With-Python.jpg",
      "alt": "How to Develop a Weighted Average Ensemble With Python",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/07/Box-and-Whisker-Plots-of-Accuracy-of-Singles-Model-Fit-On-Selected-Features-vs-Ensemble.png",
      "bytes": 6038,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/033.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/07/Box-and-Whisker-Plots-of-Accuracy-of-Singles-Model-Fit-On-Selected-Features-vs-Ensemble.png",
      "alt": "Box and Whisker Plots of Accuracy of Singles Model Fit On Selected Features vs. Ensemble",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://machinelearningmastery.com/wp-content/uploads/2020/02/Box-Plot-of-Gradient-Boosting-Ensemble-Tree-Depth-vs-Classification-Accuracy.png",
      "bytes": 8388,
      "reused": true,
      "local_path": "attachments/9152a1982f64/images/037.png",
      "src": "https://machinelearningmastery.com/wp-content/uploads/2020/02/Box-Plot-of-Gradient-Boosting-Ensemble-Tree-Depth-vs-Classification-Accuracy.png",
      "alt": "Box Plot of Gradient Boosting Ensemble Tree Depth vs. Classification Accuracy",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://machinelearningmastery.com/author/jasonb/",
      "label": "Jason Brownlee",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/category/ensemble-learning/",
      "label": "Ensemble Learning",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/random-forest-ensemble-in-python/",
      "label": "49",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/ensemble-learning-algorithms-with-python/",
      "label": "Ensemble Learning Algorithms With Python",
      "type": "a"
    },
    {
      "href": "https://flickr.com/photos/sheila_sund/30461181307/",
      "label": "Sheila Sund",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/bagging-ensemble-with-python/",
      "label": "bootstrap aggregation (bagging)",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/a-gentle-introduction-to-the-bootstrap-method/",
      "label": "bootstrap sample",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/classification-and-regression-trees-for-machine-learning/",
      "label": "classification and regression trees",
      "type": "a"
    },
    {
      "href": "https://amzn.to/2O3BuOa",
      "label": "Applied Predictive Modeling",
      "type": "a"
    },
    {
      "href": "https://amzn.to/37xa7DT",
      "label": "An Introduction to Statistical Learning with Applications in R",
      "type": "a"
    },
    {
      "href": "https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html",
      "label": "RandomForestRegressor",
      "type": "a"
    },
    {
      "href": "https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html",
      "label": "RandomForestClassifier",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/k-fold-cross-validation/",
      "label": "cross-validation",
      "type": "a"
    },
    {
      "href": "https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html",
      "label": "make_classification() function",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/cross-validation-for-imbalanced-classification/",
      "label": "repeated stratified k-fold cross-validation",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/different-results-each-time-in-machine-learning/",
      "label": "results may vary",
      "type": "a"
    },
    {
      "href": "https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_regression.html",
      "label": "make_regression() function",
      "type": "a"
    },
    {
      "href": "https://amzn.to/3ahyI1O",
      "label": "The Elements of Statistical Learning",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/implement-random-forest-scratch-python/",
      "label": "How to Implement Random Forest From Scratch in Python",
      "type": "a"
    },
    {
      "href": "https://link.springer.com/article/10.1023/A:1010933404324",
      "label": "Random Forests",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Random_forest",
      "label": "Random Forest, Wikipedia",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/random-subspace-ensemble-with-python/",
      "label": "How to Develop a Random Subspace Ensemble With Python",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/bagging-and-random-forest-ensemble-algorithms-for-machine-learning/",
      "label": "Bagging and Random Forest Ensemble Algorithms for…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/random-forest-ensembles-with-xgboost/",
      "label": "How to Develop Random Forest Ensembles With XGBoost",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/weighted-average-ensemble-with-python/",
      "label": "How to Develop a Weighted Average Ensemble With Python",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/feature-selection-subspace-ensemble-in-python/",
      "label": "How to Develop a Feature Selection Subspace Ensemble…",
      "type": "a"
    },
    {
      "href": "https://machinelearningmastery.com/gradient-boosting-machine-ensemble-in-python/",
      "label": "How to Develop a Gradient Boosting Machine Ensemble…",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:54:36Z",
  "source_url": "https://machinelearningmastery.com/random-forest-ensemble-in-python/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 44779,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9152a1982f64/pages/source.html",
  "local_html": "attachments/9152a1982f64/pages/article.html",
  "body_base_path": "attachments/9152a1982f64/pages/",
  "body_markdown_path": "attachments/9152a1982f64/pages/article.md"
}