保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

How Random Forests & Decision Trees Decide: Simply Explained With An Example In Python

What is a decision tree? What is a random forest? How do they decide to split the data? How do they predict class labels in classification... Serafeim Loukas, PhD June 1, 20228 min read

What is a decision tree? What is a random forest? How do they decide to split the data? How do they predict class labels in classification tasks? Everything explained in 9 minutes.

A visual example of a random forest consisting of 3 decision trees Source. A visual example of a random forest consisting of 3 decision trees (Source). Hi folks! I hope you are doing well. I had intented to write this post for a long time, so here it is! Let's get started! Learn this step by step with the interactive AI and Data Scientist and Machine Learning roadmaps.

Introduction

Before we can understand how Random Forests (RF) work and decide, we need to understand first what are the Decision Trees and how they work.

Decision Trees

Example 1: The ideal case

Let's assume that we have a labeled dataset with 10 samples in total. 5 of these samples belong to the dog class (blue) and the remaining 5 to the cat class (red). Also, we assume we have only 2 features/variables, thus our variable space is 2D. Here is some Python code to create the dataset and plot it: text

X , y = make_classification(n_samples=10, n_features=2, n_informative=2, n_redundant=0, n_repeated=0, random_state=0)

text

c = ["blue" if i==0 else "red" for i in y]

text

plt.figure(dpi=200)
plt.scatter(X[:,0], X[:,1], c=c)
plt.xlabel("x")
plt.ylabel("y")
plt.axvline(x=0.9, label="split at x=0.9", c = "k", linestyle="--")
plt.legend()
plt.show()

Fig 1: Example of a dataset. Figure made in python by the author. Fig 1: Example of a dataset. Figure made in python by the author. What the Decision Trees do is simple: they find ways to split the data in a way such as that separate as much as possible the samples of the classes (increasing the class separability). In the above example, the perfect split would be a split at x=0.9 as this would lead to 5 red points being at the left side and the 5 blue at the right side (perfect class separability). Each time we split the space/data like that, we actually build a decision tree with a specific rule. The decision tree would look like the following: Fig 2: The decision tree. Split at x=0.9. Figure made by the author in smartdraw. Fig 2: The decision tree. Split at x=0.9. Figure made by the author in smartdraw. Here we initially have the root node containing all the data and then, we split the data at x=0.9 leading to two branches leading to two leaf nodes. Since we cannot split the data more (we cannot add new decision nodes since the data are perfectly split), the decision tree construction ends here. No need to split the data in the second feature dimension i.e. y axis. Making a new prediciton: If we would pass a new sample (e.g., from the test set) that had a feature/variable x=5 , that would end up in the left leaf node and would classified as class "red ".

Everytime we split at a specific x or y coordinate, we create a decision node!

Example 2: A real case using the Gini Impurity

Usually, the data cannot be separated so easily and it takes a lot of effort/iterations (this is done during model training/fitting) to find the optimal splits. But the question is:

How can we find these splits using mathematical formulations? Let's look at another example now with 3 classes, each having 5 samples. Here is some Python code to create the dataset and plot it: text

X,y = make_classification(n_samples=15, n_features=2, n_informative=2, n_redundant=0, n_repeated=0, random_state=1,n_classes=3, n_clusters_per_class=1)

text

c = ["blue" if i==0 else "red" if i==1 else "green" for i in y]

text

plt.figure(dpi=200)
plt.scatter(X[:,0], X[:,1], c=c)
plt.xlabel("x")
plt.ylabel("y")
plt.axvline(x=0.9, label="split at x=0.9", c = "k", linestyle="--")
plt.axhline(y=0.4, xmin=0, xmax=0.59, label="split at y=0.4", c = "pink", linestyle="--")
plt.legend()
plt.show()

Fig 3: Example of a dataset. Figure made in python by the author. Fig 3: Example of a dataset. Figure made in python by the author. Let's use the same inital rule as before first (we split at x=0.9). Fig 4: The decision tree. Split at x=0.9. Figure made by the author in smartdraw. Fig 4: The decision tree. Split at x=0.9. Figure made by the author in smartdraw. We start again with a root node containing all the data and then, we split the data at x=0.9 leading to two branches leading to two leaf nodes. The right leaf contains all the blue samples and the left the ref and green samples. Here, we cannot stop and we need to move forward one more step. We will now split the data in the second feature dimension i.e. y axis to try to reparate the green dots from the reds. The decison tree becomes (for a y=0.4 split): Fig 5: The decision tree. Second Split at y=0.4. Figure made by the author in smartdraw. Fig 5: The decision tree. Second Split at y=0.4. Figure made by the author in smartdraw. Making a new prediciton: If we would pass a new sample (e.g., from the test set) that had a features/variables x=5 and y=-2 , that would end up in the left leaf node and would classified as class "green ".

Reminder: Everytime we split at a specific x or y coordinate, we create a decision node!

The Gini Impurity & Gini Gain

The principle idea behind Gini Impurity is simple:

Gini Impurity tells us what is the probability that we classify the datapoint wrongly [3]. In the above example, we found the perfect split levels via visual inspection. However, in real world problems, this is not possible because the number of samples and features are large. To find the optimal split levels (in each feature dimension), decision trees choose the split that maximizes the Gini Gain [3].

The best possible Gini Impurity score is 0 and it corresponds to the optimal split at the current level/state of the model. The mathematical formualtion is as follows. For C classes in our data, p(i) being the probability of choosing a data sample that belongs to class i , the Gini Impurity (G) is given by: Formula made in markdown by the author. Formula made in markdown by the author. Now, let's go back to our first (most simple) example (Example 1), define a new bad split and estimate the Gini Impurity and gain step by step! First, let's estimate p(i) for each class (without a split yet):

Now, let's estimate the initial G (without any split yet):

(50% chance to wrongly classify it since we have 4 possible events and only 2 are correct i.e. choose blue, classify blue , choose blue, classify red, choose red, classify blue, choose red, classify red). Fig 6: Example of a dataset. Figure made in python by the author. Fig 6: Example of a dataset. Figure made in python by the author. Now, let's estimate the the Gini Impurity for the x=0.2 "non-perfect" split.

The final Gini Gain is computed by subtracting the weighted impurities (G_left & G_right) from the original impurity G0 (no split):

Reminder: The higher the Gini Gain the better the split. To conclude, the decision tree algorithm, tries a lot of different splits, it estimates the Gini Gain of each split and splits acocrding to the maximum Gini Gain. So, we end up optimal splits w.r.t. the Gini Gain.

Task : Try to estimate now the Gini gain for the perfect split as shown in Fig 1.

Tip: You will easily find G0 = 0.5 and G_left & G_right = 0, thus a Gini Gain of 0.5-(5/10)0-(5/10) 0 = 0.5!

Random Forests

Now that you understand that a desicion tree is and how it finds the best splits (based on the Gini Gain), it is time to introduce the Random Forests. Random forests are nothing more than an ensemble of decision trees [1]. One important thing to notice here is that random forest usually use a subsample of the available variables / features instead of all of them, for each unique decision tree in the forest. E.g. see input argument "max_features " in https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html model. Finally, random forests usually decide by taking the majority voting or the average predicted class of the individual decison trees that constitute the forest.

Conclusions

Decision trees and random forests are powerful machine learning models that can be used for regression and classification. In the case of classification problems, the best splits are done based on the Gini Gain score as explained above. In the near future, I will write a python-only article showing how to fit a model using sklearn and real data. That's all folks! Hope you liked this article. Any questions? Post them as a comment and I will reply as soon as possible.

•NEW : After a great deal of hard work and staying behind the scenes for quite a while, we're excited to now offer our expertise through a platform, the "Data Science Hub" on Patreon (https://www.patreon.com/TheDataScienceHub). This hub is our way of providing you with bespoke consulting services and comprehensive responses to all your inquiries , ranging from Machine Learning to strategic data analytics planning. Also, make sure to subscribe to my mailing list!

Stay tuned & support this effort

If you liked and found this article useful, follow me and make sure you subscribe to my mailing list and become a member:

References

[1] https://en.wikipedia.org/wiki/Random_forest [2] https://www.smartdraw.com/ [3] https://en.wikipedia.org/wiki/Decision_tree_learning

Get in touch with me

You may also like:

Support Vector Machines (SVM) clearly explained: A python tutorial for classification problems... K-Means Clustering: How It Works & Finding The Optimum Number Of Clusters In The Data LSTM Time-Series Forecasting: Predicting Stock Prices Using An LSTM Model Time-Series Forecasting: Predicting Stock Prices Using An ARIMA Model The Best FREE Data Science Resources: FREE Books & Online Courses ROC Curve Explained using a COVID-19 hypothetical example: Binary & Multi-Class Classification... Support Vector Machines (SVM) clearly explained: A python tutorial for classification problems... PCA clearly explained - How, when, why to use it and feature importance: A guide in Python Everything you need to know about Min-Max normalization in Python How Scikit-Learn's StandardScaler works

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Serafeim Loukas, PhD

See all from Serafeim Loukas, PhD Data ScienceDecision TreeMachine LearningPythonRandom Forest Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

8 Revolutionary Artificial Intelligence Technologies Of The Modern Era!

Artificial Intelligence Exploring and analyzing the 8 AI Technologies that will or already have influenced the world drastically! Bharath K January 20, 202117 min read

Recursion vs Dynamic Programming - Fibonacci(Leetcode 509)

In this blog, I will use Leetcode 509. Fibonacci Number as our example to illustrate the coding logic and complexity of recursion vs... Shuheng.Ma October 3, 20216 min read

Graphs 101: Airline Transportation Network

Machine Learning A hands-on overview with a real world dataset Bruno Gonçalves July 13, 20217 min read

Toronto on Fire in Data, Part 1

Data Science Fire Incidents Analysis by Segmentation & Poisson in Practice Anders Ohrn February 6, 20199 min read

Simple Versioned Datasets With Github Actions

Programming In Less Than 10 Lines Of Python - Preserving The History of Notable CEOs Dylan Roy October 12, 20206 min read

Entropy based Uncertainty Prediction

Artificial Intelligence This article explores how Entropy can be employed as a tool for uncertainty estimation in image segmentation tasks. We will walk through... François Porcher September 2, 20236 min read

4 Principles to Guide Data Team Leaders

Data Science A management philosophy for building high-performing teams Mark Derdzinski August 8, 20229 min read

Mastering Statistics with Pandas

Programming Using Pandas to Generate Statistics Sadrach Pierre, Ph.D. July 3, 20203 min read

Make Your Art Move with Stable Diffusion Animations

Deep Learning Introducing Giffusion, a simple web UI to create animated GIFs and Videos with Stable Diffusion Dhruv Nair December 19, 20229 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close