保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

When Predictors Collide: Mastering VIF in Multicollinear Regression

Explore how the Variance Inflation Factor helps detect and manage multicollinearity in your regression models. Niklas Lang April 16, 20259 min read Photo by MARIOLA GROBELSKA via Unsplash In regression models, the independent variables must be not or only slightly dependent on each other, i.e. that they are not correlated. However, if such a dependency exists, this is referred to as multicollinearity and leads to unstable models and results that are difficult to interpret. The variance inflation factor is a decisive metric for recognizing multicollinearity and indicates the extent to which the correlation with other predictors increases the variance of a regression coefficient. A high value of this metric indicates a high correlation of the variable with other independent variables in the model. In the following article, we look in detail at multicollinearity and the VIF as a measurement tool. We also show how the VIF can be interpreted and what measures can be taken to reduce it. We also compare the indicator with other methods for measuring multicollinearity.

What is Multicollinearity?

Multicollinearity is a phenomenon that occurs in regression analysis when two or more variables are strongly correlated with each other so that a change in one variable leads to a change in the other variable. As a result, the development of an independent variable can be predicted completely or at least partially by another variable. This complicates the prediction of linear regression to determine the influence of an independent variable on the dependent variable. A distinction can be made between two types of multicollinearity: Learn this step by step with the interactive AI and Data Scientist roadmap.

The occurrence of multicollinearity in regressions leads to serious problems as, for example, the regression coefficients become unstable and react very strongly to new data, so that the overall prediction quality suffers. Various methods can be used to recognize multicollinearity, such as the correlation matrix or the variance inflation factor, which we will look at in more detail in the next section.

What is the Variance Inflation Factor (VIF)?

The variance inflation factor (VIF) describes a diagnostic tool for regression models that helps to detect multicollinearity. It indicates the factor by which the variance of a coefficient increases due to the correlation with other variables. A high VIF value indicates a strong multicollinearity of the variable with other independent variables. This negatively influences the regression coefficient estimate and results in high standard errors. It is therefore important to calculate the VIF so that multicollinearity is recognized at an early stage and countermeasures can be taken. The VIF for a single variable iii is calculated using this formula: [latex] \[VIF = \frac{1}{(1 - R^2)}\] Here R2R^2R2 is the so-called coefficient of determination of the regression of feature iii against all other independent variables. A high R2R^2R2 value indicates that a large proportion of the variables can be explained by the other features, so that multicollinearity is suspected. In a regression with the three independent variables X1X_1X1​, X2X_2X2​ and X3X_3X3​, for example, one would train a regression with X1X_1X1​ as the dependent variable and X2X_2X2​ and X3X_3X3​ as independent variables. With the help of this model, R12R_{1}^2R12​ could then be calculated and inserted into the formula for the VIF. This procedure would then be repeated for the remaining combinations of the three independent variables. A typical threshold value is VIF > 10, which indicates strong multicollinearity. In the following section, we look in more detail at the interpretation of the variance inflation factor.

How can different Values of the Variance Inflation Factor be interpreted?

After calculating the VIF, it is important to be able to evaluate what statement the value makes about the situation in the model and to be able to deduce whether measures are necessary. The values can be interpreted as follows:

Overall, a high VIF value indicates that the variable may be redundant, as it is highly correlated with other variables. In such cases, various measures should be taken to reduce multicollinearity.

What measures help to reduce the VIF?

There are various ways to circumvent the effects of multicollinearity and thus also reduce the variance inflation factor. The most popular measures include:

These methods can be used to effectively reduce the VIF and combat multicollinearity in a regression. This makes the results of the model more stable and the standard error can be better controlled.

How does the VIF compare to other methods?

The variance inflation factor is a widely used technique to measure multicollinearity in a data set. However, other methods can offer specific advantages and disadvantages compared to the VIF, depending on the application.

Correlation Matrix

The correlation matrix is a statistical method for quantifying and comparing the relationships between different variables in a data set. The pairwise correlations between all combinations of two variables are shown in a tabular structure. Each cell in the matrix contains the so-called correlation coefficient between the two variables defined in the column and the row. This value can be between -1 and 1 and provides information on how the two variables relate to each other. A positive value indicates a positive correlation, meaning that an increase in one variable leads to an increase in the other variable. The exact value of the correlation coefficient provides information on how strongly the variables move about each other. With a negative correlation coefficient, the variables move in opposite directions, meaning that an increase in one variable leads to a decrease in the other variable. Finally, a coefficient of 0 indicates that there is no correlation. Example of a Correlation Matrix | Source: Author A correlation matrix therefore fulfills the purpose of presenting the correlations in a data set in a quick and easy-to-understand way and thus forms the basis for subsequent steps, such as model selection. This makes it possible, for example, to recognize multicollinearity, which can cause problems with regression models, as the parameters to be learned are distorted. Compared to the VIF, the correlation matrix only offers a surface analysis of the correlations between variables. However, the biggest difference is that the correlation matrix only shows the pairwise comparisons between variables and not the simultaneous effects between several variables. In addition, the VIF is more useful for quantifying exactly how much multicollinearity affects the estimate of the coefficients.

Eigenvalue Decomposition

Eigenvalue decomposition is a method that builds on the correlation matrix and mathematically helps to identify multicollinearity. Either the correlation matrix or the covariance matrix can be used. In general, small eigenvalues indicate a stronger, linear dependency between the variables and are therefore a sign of multicollinearity. Compared to the VIF, the eigenvalue decomposition offers a deeper mathematical analysis and can in some cases also help to detect multicollinearity that would have remained hidden by the VIF. However, this method is much more complex and difficult to interpret. The VIF is a simple and easy-to-understand method for detecting multicollinearity. Compared to other methods, it performs well because it allows a precise and direct analysis that is at the level of the individual variables.

How to detect Multicollinearity in Python?

Recognizing multicollinearity is a crucial step in data preprocessing in machine learning to train a model that is as meaningful and robust as possible. In this section, we therefore take a closer look at how the VIF can be calculated in Python and how the correlation matrix is created.

Calculating the Variance Inflation Factor in Python

The variance inflation factor can be easily used and imported in Python via the statsmodels library. Assuming we already have a Pandas DataFrame in a variable X that contains the independent variables, we can simply create a new, empty DataFrame for calculating the VIFs. The variable names and values are then saved in this frame. Collinearity A new row is created for each independent variable in X in the Variable column. It is then iterated through all variables in the data set and the variance inflation factor is calculated for the values of the variables and again saved in a list. This list is then stored as column VIF in the DataFrame.

Calculating the Correlation Matrix

In Python, a correlation matrix can be easily calculated using Pandas and then visualized as a heatmap using Seaborn. To illustrate this, we generate random data using NumPy and store it in a DataFrame. As soon as the data is stored in a DataFrame, the correlation matrix can be created using the corr() function. If no parameters are defined within the function, the Pearson coefficient is used by default to calculate the correlation matrix. Otherwise, you can also define a different correlation coefficient using the method parameter. Correlation Matrix / Korrelationsmatrix Finally, the heatmap is visualized using seaborn. To do this, the heatmap() function is called and the correlation matrix is passed. Among other things, the parameters can be used to determine whether the labels should be added and the color palette can be specified. The diagram is then displayed with the help of matplolib.

This is what you should take with you

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Niklas Lang

See all from Niklas Lang Editor's PicksMulticollinearityPythonStatisticsVariance Inflation Factor Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

A Series on Flask APIs, Part 1: GETting and POSTing

Programming A how-to on GETting and POSTing data to a Flask API Alexis McKenzie August 23, 20214 min read

Web scraping with Scrapy: Theoretical Understanding

This is the first part of a 4 part tutorial series on web scraping using Scrapy and Selenium. Karthikeyan P July 31, 202014 min read

Understand the Workings of Black-Box models with LIME

Artificial Intelligence While the performance of a machine learning model can seem impressive, it might not be making a significant impact to the business unless... Suhas Maddali August 3, 20228 min read

Nearest Neighbor Analysis for Large Datasets in Helsinki Region

Data Science BallTree function for efficient distance analysis: Bus stops and restaurants Bryan Vallejo December 29, 20217 min read

Digging Into My Year of Spotify Streaming with Track Embeddings

Machine Learning I take a high-level look at my year of music streaming Mads Kalør January 12, 20229 min read

Custom Memory for ChatGPT API

Artificial Intelligence A Gentle Introduction to LangChain Memory Types Andrea Valenzuela August 20, 20239 min read

How to Build an Impressive Data Science Resume?

Machine Learning Tips to make an impactful resume that stands out Sharan Kumar Ravindran April 17, 20217 min read

Optimising Non-Linear Treatment Effects in Pricing and Promotions

Machine Learning Causal AI, exploring the integration of causal reasoning into machine learning Ryan OSullivan May 24, 202412 min read

Step by Step Basics: Text Classifier

NLP An Instructional Guide and Flow Diagram for Building a Supervised Machine Learning Text Classifier in Python Lucy Dickinson February 17, 202311 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close