保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Data Science

Model Drift Introduction and Concepts

Learn some of the concepts behind machine learning models drift and understand why MLOps is so important in today's world Ivo Bernardo June 22, 20246 min read Models change because the world changes - Image by arptrastogi at Unsplash.com Models change because the world changes - Image by arptrastogi at Unsplash.com Taxes, death and model drift are the only three certainties in life. Ok, I might have added this last one to the adage but the truth is that all models suffer from decay. After developing a machine learning mode, you will always see the same pattern develop:

After a couple years, there's a high likelihood that your model's performance is way worse than when you first developed it. This can happen for a multitude of reasons, but the fundamental cause is that the world changes. When the world changes, the data we use to represent real-life information also changes. The underlying data distributions shift, which will inevitably impact how our machine learning models learn and perform. Learn this step by step with the interactive AI and Data Scientist and Machine Learning roadmaps. In this blog post, we'll examine examples of situations where underlying changes in the world can impact your models. Understanding these examples will better prepare you to develop plans and explain to your leadership why machine learning operations (MLOps) are so important today, defining the success of a data-driven organization.

Data Distribution Shift

Data distribution shift is the most common cause of model drift. This occurs when the distribution of one of your variables shifts significantly compared to the time of development. Let's assume that we have a model that uses the age of a user as a feature for predicting app churn[1] with the following distribution: Age Distribution at Time t - Image by Author Age Distribution at Time t - Image by Author [1] Churn is the act of a customer deciding to stop using a service, voluntarily Time t is when you developed the model. Now, imagine that the age distribution of the user base you are studying shifts to something like the following after a year: Age Distribution at Time t vs. t+1— Image by Author Age Distribution at Time t vs. t+1— Image by Author The user base is now much older than it was at time t. This inherently leads to worse performance by your algorithm. Your model is accustomed to a certain data distribution that no longer reflects current reality. If the shift is too large, you might end up predicting on examples for which you had a very small sample size during training. This effect gets even worse if you use models that are highly dependent on feature distribution (for example, tree-based models use distribution sample size to make their cutoff on the branches). Now, consider models that have hundreds of features where every feature will suffer from distribution shift - the impact on your models is enormous. Catching data distribution shift is relatively easy:

Monitoring tools such as MLFlow or Azure Machine Learning Studio will help you keep track of your features' data distribution. Using these tools, it's quite normal to automatically retrain models with triggers based on certain distribution shifts.

Edge Cases

Another common situation that affects machine learning models in production is the issue of edge cases. For example, let's imagine that you are forecasting sales for a specific product. Ideally, you would like the world to remain static, with the factors affecting the sales of your product staying constant. However, edge cases may suddenly arise and undermine your entire model. An abnormally low or high value in your sales can break your predictions. This happened during the COVID-19 pandemic, where edge cases started to become the norm, rendering most machine learning models useless. Although this is a very obvious edge case situation, sometimes edge cases are trickier to spot. Typically, it requires a deep understanding of the underlying phenomena we are working with. Nevertheless, there are a couple of actions you can take:

However, looking at outliers shouldn't be the only way to spot edge cases. The "difficult" edge cases to identify are normally not outliers and need to be understood with the help of business users who understand the context of the underlying data.

Feedback Loops

In certain scenarios, your model acts on features of other models or even on itself, as in the case of time-series models. Typically, if you have model outputs that work as features, this will produce some level of feedback loop. Checking the output values of your models and keeping track of their distribution should be enough to understand the likelihood of producing a feedback loop. Particularly, it's important to measure explainability metrics such as SHAP values to understand if the influence of features generated by models is changing over time. Ideally, you should try to avoid involving any feature that is produced or can be influenced by the output of a machine learning model. If you can't avoid this due to certain business or technical decisions, then monitoring becomes absolutely critical.

Missing-as-a-Feature

In model development pipelines, we typically perform a step related to missing value imputation, enabling your model to deal with non-existent information. During model development, it is common practice to examine the number of missing values in your features and address them only if the proportion of missing data in a column falls below a specified threshold, discarding the feature if it's over that value. A common problem arises when you put your model into production and one of the features starts receiving more missing values than expected. What will happen? You will start to predict based mainly on missing information (and your imputation method)! This is a pattern commonly called "missing-as-a-feature " Essentially, you are just using your imputation method as values for the features in your predictions. Missing as a Feature Example - Image by Author Missing as a Feature Example - Image by Author Notice that tracking distribution is not enough to spot missing-as-feature scenarios. You need to explicitly monitor the % of missing values going into your model before the imputation step in your pipeline.

That's it! Thank you for taking the time to read this post. Model drift is a recent area where a lot of research is actively being developed - namely, how to use automatic triggers to retrain your model, or understand if you should train a new model from scratch. There are many tools available that will help you manage your post-production models more effectively. The following is not an exhaustive list but contains some of the most well-known frameworks that you can experiment with:

See you on the next post! Feel free to visit me on my youtube channel, Udemy profile, or substack:

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Ivo Bernardo

See all from Ivo Bernardo Data ScienceMachine LearningMlflowMlops Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

NODE: Tabular-Focused Neural Trees

Artificial Intelligence Exploring NODE: A Neural Decision Tree architecture for tabular data Nakul Upadhya July 4, 20237 min read

What Has Machine Learning Been Up To In Healthcare?

Deep Learning A High-Level Look at What's Been Done Hien Vo, M.S. October 5, 20214 min read

Solving Combinatorial Problems with PySpark

Data Science Partitioning combinatorial problems using binary representation Salil Jain August 4, 20204 min read

What's in a Tuple? - Part 2

Programming Now that you've learned about tuples in Python, I'll walk through an example that shows why they're useful. Murtaza Ali February 2, 20223 min read

8 Machine Learning Algorithms Everyone New to Data Science Should Know

Machine Learning Shortly Explaining the Algorithms Behind Machine Learning Niklas Lang October 4, 202212 min read

Coding Bootcamps -What are they Like?

Programming My personal experience attending a coding bootcam Marco Santos August 21, 202110 min read

Detecting Mask On/Off Or Incorrectly Worn Using Yolo-v4

Machine Learning The past two years has been surreal to say the least for most of us. These days you look around and regardless of which part of the world... Tan Kah Ming November 16, 20216 min read

Don't Crash Your App: Load Records from the Database in Batches for Better Performance

Programming Save your Python app's performance by efficiently loading query Mike Huls April 18, 20248 min read

How to make a PyTorch Transformer for time series forecasting

Deep Learning This post will show you how to transform a time series Transformer architecture diagram into PyTorch code step by step. Kasper Groes Albin Ludvigsen May 12, 202211 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network

Recommended Articles

Close