保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Regression in the face of messy outliers? Try Huber regressor

Outliers in the data are ubiquitous, and they can mess up your regression problem. Try Hubber regressor to tackle this problem. Tirthajyoti Sarkar November 4, 20214 min read Image source: Created by the author Image source: Created by the author

What's the problem?

Let's say you have a dataset with two features _X_1 and _X_2, on which you are performing linear regression. However, some noise/outliers got introduced in the dataset. Say, the y-value outliers are exceptionally low as compared to what they should be. How does that look like? This is the data you got. Image source: Created by the author Image source: Created by the author However, if you really think about the slope of the X -Y data, the expected y-values should have been much higher for those X -values. Something like the following, Learn this step by step with the interactive AI and Data Scientist, Computer Science and Machine Learning roadmaps. Image source: Created by the author Image source: Created by the author These are obvious outliers and you can run a simple exploratory data analysis (EDA) to catch and discard them from the dataset before building the regression model. But you cannot hope to catch all the outliers - at scale and in all dimensions. Visualization of a dataset with 100 or 1000 dimensions (features) is challenging enough to manually examine the plots and discover outliers. A regression algorithm that is robust to outliers sounds like a good bet against those pesky bad data points. The Huber regressor is one such tool that we will discuss in this article.

Huber regression

In statistics, Huber loss is a particular loss function (first introduced in 1964 by Peter Jost Huber, a Swiss mathematician) that is used widely for robust regression problems - situations where outliers are present that can degrade the performance and accuracy of least-squared-loss error based regression.

Where did the least-square come from?

Technical details

The loss is given by, Image source: Wikipedia Image source: Wikipedia We can see that the loss is the square of the usual residual (y - f(x)) only when the absolute value of the residual is smaller than a fixed parameter. The choice and tuning of this parameter are important to get a good estimator. Where the residual is greater than this parameter, the loss is a function of the absolute value of the residual and the Huber parameter. Now, you may remember from elementary statistics that the squared loss comes from the unbiased estimator around the mean whereas an absolute difference loss comes from an unbiased estimator around the median. Median is much more robust to outliers than mean. Huber loss is a balanced compromise between these two types. It is robust to the outliers but does not completely ignore them either. The tuning can be done with the free parameter, of course. Image source: Created by the author Image source: Created by the author

A Python demo

The demo notebook is here in my Github repo. We created the synthetic data and added some noisy outlier with the following code, text

import numpy as np
from sklearn.datasets import make_regression

text

rng = np.random.RandomState(0)
X, y, coef = make_regression(
    n_samples=200, n_features=2, noise=4.0, coef=True, random_state=0)

text

# The first four data points are outlier
X[:4] = rng.uniform(10, 20, (4, 2))
y[:4] = rng.uniform(10, 20, 4)

Now, all you have to do is to call Scikit-learn's built-in HuberRegressor estimator and fit the data. For comparison, we also have the standard LinearRegression method called in. text

from sklearn.linear_model import HuberRegressor, LinearRegression

text

huber = HuberRegressor().fit(X, y)
linear = LinearRegression().fit(X, y)

Now, we know that the first 4 data points are outliers. So, if we try to predict the y-value with the first data point, we will get something like this, text

linear.predict(X[:1,])

text

>> array([87.38004436])

As expected, the linear regression prediction is a low value for y. Why? Because the linear fit (based on least-squared loss) has bent towards the outliers due to their large leverage. Image source: Created by the author Image source: Created by the author However, the Huber estimator predicts a more reasonable (high) value, text

huber.predict(X[:1,])

text

>> array([806.72000092])

To demonstrate the robustness of the Huber estimator further, we can use the estimated coefficients and plot the best-fitted line, text

huber_y1 = np.arange(-2.5,20,0.01)*huber.coef_[0] + 
            np.arange(-2.5,20,0.01)*huber.coef_[1] + 
            huber.intercept_

text

plt.figure(dpi=120)
plt.scatter(X[:,0],y)
plt.plot(np.arange(-2.5,20,0.01), 
         huber_y1,
         color='red',linestyle='--')
plt.show()

Image source: Created by the author Image source: Created by the author

Theil-Sen estimator

Although we are not discussing it in this article, readers are encouraged to check the Theil-Sen estimator, which is another robust linear regression technique and enjoys being highly insensitive to outliers. As expected, Scikit-learn has a built-in method for this estimator too: Scikit-learn Theil-Sen estimator. For multiple linear regression with a large number of features, this is a very efficient and fast regression algorithm among all the robust estimators.

Summary

In this brief article, we talked about the problem of linear regression estimators in the presence of outliers in the dataset. We demonstrated, with a simple example, that linear estimators based on traditional least-squared loss function, may predict completely wrong values because they are bent towards the outliers. We discussed a couple of robust estimators and demonstrated the Huber regressor in detail. Non-parametric statistics use these robust regression techniques in many places, especially when the data is expected to be particularly noisy. Data science students and professionals alike should also have a working knowledge of these robust regression methods for automating the modeling of large datasets in the presence of outliers.

Loved the article? Become a Medium member to continue learning without limits . I'll receive a portion of your membership fee if you use the following link,__with no extra cost to you .

Join Medium with my referral link - Tirthajyoti Sarkar

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Tirthajyoti Sarkar

See all from Tirthajyoti Sarkar Data ScienceMachine LearningProgrammingPython Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

Google Analytics is Digital Marketing, but Digital Marketing is Not Analytics

Data Science These are disciplines that apply different methods and tools to perform their analyses, but above all with very different objectives Jesus Fajardo June 14, 20217 min read

Enhancing Visualization of Well Logs With Plot Fills

Data Visualization Applying Color Infill to Well Log Data Using matplotlib Andy McDonald January 27, 20212 min read

Machine Learning in Academic Research v.s. Practical

Artificial Intelligence Development Phases of a typical industrial machine learning project Christopher Tao February 19, 20208 min read

Coding consciousness

Artificial Intelligence A very long post about the Turing Test Massimo Belloni September 20, 201915 min read

Spider and Parallel Charts in R with the ggvanced Package

Data Science An R package for effective visualization of multiple variables Zvonimir Boban June 20, 20235 min read

How to Style Pandas DataFrames Like a Pro

Data Science Make your DataFrames stunning with this complete guide for beginners. Dario Radečić July 27, 20228 min read

Prediction Intervals for any Regression Model

Machine Learning Conformal Prediction in Julia - Part 3 Patrick Altmeyer December 12, 20228 min read

Using Permutation Tests to proof the Climate Change

Data Science A simple statistical test shows that average temperatures are very unlikely to increase due to "bad luck". Kaya Kupferschmidt December 23, 20209 min read

3 Examples That Show The Unlimited Flexibility of PySpark

Artificial Intelligence A combination of Python and SQL but easier than both Spark is an analytics engine... Soner Yıldırım June 5, 20215 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close