{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Chip Huyen",
  "title_listed": "Real-time machine learning: challenges and solutions",
  "keywords": "实时ML,流处理,特征",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huyenchip.com/2022/01/02/real-time-machine-learning-challenges-and-solutions.html",
  "publish_date": "2022-01",
  "corpus_stage": "推理部署",
  "notes": "实时机器学习挑战与方案",
  "id": "e61555102aed",
  "collected_at": "2026-10-02T14:14:19",
  "title": "Real-time machine learning: challenges and solutions",
  "body_markdown": "[\nTwitter discussion\n,\nLinkedIn\n]\nUpdates\nJan 3, 2023: Update the online features section to differentiate between real-time features and near real-time features.\nIf you’re interested in this topic, my book\nDesigning Machine Learning Systems\n(O’Reilly, June 2022) covers online prediction and continual learning in much more detail.\nReal-time machine learning is the approach of using real-time data to generate more accurate predictions and adapt models to changing environments. A year ago, I wrote a post on how\nmachine learning is going real-time\n. The post must have captured many data scientists’ pain points because, after the post, many companies reached out to me sharing their pain points and discussing how to move their pipelines real time. These conversations prompted me to start a company on real-time machine learning. We have some exciting news I can’t wait to share with you, but that’s a story for another time :-)\nIn the last year, I’ve talked to ~30 companies in different industries about their challenges with real-time machine learning. I’ve also worked with quite a few to find the solutions. This post outlines the solutions for (1) online prediction and (2) continual learning, with step-by-step use cases, considerations, and technologies required for each level.\nDepending on your experience, some stages might seem basic to you. It’s ok to skip those!\nDisclaimer\n: the perspective outlined in this post is heavily influenced by the companies I’ve talked to/worked with, which are predominantly tech companies. Feedback & suggestions are much appreciated.\nI’d also love to learn more about the ML workflows at your company for a more accurate perspective. Here’s the link to a\n5-minute survey\n. The results will be aggregated, summarized, and shared with the community. Thank you!\nTowards Online Prediction\nWhile I believe that we’re still a few years away from mainstream adoption of continual learning, I’m seeing significant investments from companies to move towards online inference. Starting with a batch prediction system, we’ll discuss what’s needed for a simple online prediction system using batch features, typically useful for in-session adaptation (e.g. giving a user predictions based on their activities within a session on a website or mobile app). Then we’ll continue to discuss how to move to a more mature online prediction system that leverages both complex streaming + batch features.\nStage 1. Batch prediction\nAt this stage, all predictions are precomputed in batch, generated at a certain interval, e.g. every 4 hours or every day. Typical use cases for batch prediction are collaborative filtering, content-based recommendations. Examples of companies that use batch prediction are DoorDash’s restaurant recommendations, Reddit’s subreddit recommendations, Netflix’s recommendations circa 2021. As of this post, Netflix is moving their predictions online.\nBatch prediction has many limitations, as outlined in my\nprevious post\n. Here, I want to quickly go over one example. Consider an e-commerce website where half of their visitors are new users or aren’t logged in. Because these visitors are new, there are no precomputed recommendations personalized to them. By the time the next batch of recommendations is generated, these visitors might have already left without making a purchase because they didn’t find anything relevant to them.\nA typical batch prediction workflow\nBatch prediction is NOT a prerequisite for online prediction. Batch prediction is largely a product of legacy systems.\nIn the last decade, big data processing has been dominated by batch systems like MapReduce and Spark, which allow us to periodically process a large amount of data very efficiently. When companies started with machine learning, they leveraged their existing batch systems to make predictions.\nIf you’re building a new ML system today, it’s possible to start with online prediction.\nStage 2. Online prediction with batch features\nInstead of generating predictions before requests arrive, companies in this stage generate predictions after requests arrive. They collect users’ activities on their apps in real-time. However, these events are only used to look up pre-computed embeddings to generate session embeddings. No features are computed in real-time from streaming data.\nConsider the same e-commerce website example in the previous stage. When a new visitor visits your website, instead of suggesting them generic items, you show them items based on their activities. For example, if they have looked at a keyboard and a computer monitor, they’re likely looking at work-from-home setups and your algorithm should recommend relevant items like HDMI cables or monitor mounts.\nTo do so, you need to collect and process this visitor’s activities as they happen. If this visitor has looked at item 1, item 10, and item 20, you pull out embeddings for items 1, 10, and 20 from your data warehouse. These embeddings are combined (e.g. averaged) to create this visitor’s current session embedding.\nYou want to find the most relevant items given this session embedding. Naively, you can have a model to rank every item available on your site and show the visitor the items with the highest scores. However, you might have millions of items on your site, and scoring all of them might take too long – you don’t want your visitors to wait forever to see recommendations. Most companies use another algorithm, such as item-item collaborative filtering and k-nearest neighbors, to generate a small number of candidate items (e.g. 1000) to score. This process of generating candidates is called “candidate generation” or “retrieval”. For readers interested, check out Eugene Yan’s primer on\nsession-based recommendations\nand\nGoogle’s free crash course\n.\nAn example pipeline for session-based recommendation\nThe embeddings can be learned separately from or together with the ranking model. If separately, your system will consist of at least three models: one for learning embeddings, one for retrieval, and one for ranking.\nHere, we used a recommender system to illustrate this stage, but it can be applied to other tasks including ads CTR, search, or any retrieval task.\nThe goal of session-based predictions is to increase conversion (e.g. converting first-time visitors to new users, click-through rates) and retention.\nThe list of companies that are already doing online inference or have online inference on their 2022 roadmaps is growing, including Netflix, YouTube, Roblox, Coveo, etc. Every single company that’s moved to online inference told me that they’re very happy with their metrics wins. I expect that in the next two years, most recommender systems will be session-based: every click, every view, every transaction will be used to generate fresh, relevant recommendations in (near) real-time.\nRequirements\nFor this stage, you will need to:\nUpdate your models from batch prediction to session-based predictions\n.\nThis means that you might need to add new models. Responsible team: data science/ML.\nIntegrate session data into your prediction service\n.\nTypically, you can do this with streaming infrastructure, which consists of two components:\na streaming transport\n, e.g. Kafka / AWS Kinesis / GCP Dataflow, to move streaming data (users’ activities). Most companies use managed real-time transport – it’s a pain self-host Kafka.\na streaming computation engine\n, e.g. Flink SQL, KSQL, Spark Streaming, to process streaming data. In the case of in-session adaption, this streaming computation engine is responsible for dividing users’ activities into sessions and keeping track of the information within each session (state keeping). Of the three streaming computation engines mentioned here, Flink SQL and KSQL are more recognized in the industry and provide a nice SQL abstraction for data scientists.\nMany people believe that online prediction is less efficient, both in terms of cost and performance than batch prediction because processing predictions in batch is more efficient than processing predictions one by one. This is not necessarily true, as discussed\nin the appendix\n.\nWith online prediction, you don’t have to generate predictions for users who aren’t visiting your site.\nImagine you run an app where only 2% of your users log in daily – e.g. in 2020, Grubhub had\n31 million users\nand\n622,000 daily orders\n.\nIf you generate predictions for every user each day, the compute used to generate 98% of your predictions will be wasted.\nIf your company already uses streaming for logging, this change shouldn’t be too steep. However, this might impose heavier workloads on your streaming infrastructure, which might require upgrading for it to be more efficient/scalable.\nResponsible team: data/ML platform.\nSide note\n:\nA small subset of people I’ve talked to use “streaming prediction” to refer to systems that leverage streaming infrastructure for predictions and “online prediction” to refer to systems that don’t. In this post, “online prediction” encompasses “streaming prediction”.\nChallenges\nThe challenges of this stage will be in:\nInference latency\n: with batch prediction, you don’t need to worry about the inference latency. With online prediction, however, inference latency is crucial.\nSetting up the streaming infrastructure\n: Many engineers are still terrified of doing\nSQL-like joins\non streaming even though tooling around it is maturing.\nHaving high-quality embeddings\n, especially if you deal with different item types.\nStage 3. Online prediction with online features\nBatch features\nare features extracted from historical data, often with batch processing. Also called static features or historical features.\nOnline features\nare features extracted from fresh data. Also called dynamic features.\nDepending on how online features are computed, they can be further categorized into one of the two categories: real-time (RT) features and near real-time features.\nReal-time features\nReal-time features are features computed in real-time, as soon as a prediction request is generated. Say, if you want to compute the number of views your product has had in the last 30 minutes in real-time, here are two of several simple ways you can do so:\nCreate a lambda function that takes in all the recent user activities and counts the number of views.\nStore all the user activities in a database like postgres and write a SQL query to retrieve this count.\nBecause features are computed at prediction time, they are fresh, e.g. in the order of milliseconds.\nWhile real-time features are easy to set up, they’re harder to scale. Because real-time features are computed upon receiving prediction requests, their computation latency adds directly to user-facing latency. Traffic growth and fluctuation can significantly affect computation latency. Complex features might be infeasible to support since they might take too long to compute.\nNear RT features\nLike batch features, near RT features are also precomputed and at prediction time, the latest values are retrieved and used. Since features are computed async, feature computation latency doesn’t add to user-facing latency. You can use as many features or as complex features as they want.\nUnlike batch features,\nnear RT feature values are recomputed much more frequently, so feature staleness can be in the order of seconds\n(it’s only “near” instead of “completely” real-time.) If a user doesn’t visit your site, its feature values won’t be recomputed, avoiding wasted computation.\nNear RT features are computed using a streaming processing engine.\nIf companies at stage 2 require little stream processing, companies at stage 3 might benefit much more from streaming. For example, after a user puts in order on Doordash, they might need the following features to estimate the delivery time:\nBatch features\n: the mean preparation time of this restaurant in the past\nRT features\n: how far is the restaurant location from the delivery location\nStreaming features\n: how many other orders this restaurant currently has, how many dashers will be available in the next 30 minutes\nIn the case of session-based recommendation discussed in stage 2, instead of just using item embeddings to create session embedding, you might use online features such as how long the user has spent on the site, the number of purchases an item has had in the last 24 hours.\nOnline prediction with streaming features and batch features\nExamples of companies at this stage include Stripe, Uber, Faire for use cases like\nfraud detection, credit scoring, estimation for driving and delivery\n, and\nrecommendations\n.\nThe number of online features for each prediction can be in the hundreds, if not thousands. Online features can be arbitrarily complex with joins, aggregations over large windows, transformations that depend on other transformations. Architecting a system to compute online features efficiently and affordably is very hard.\nRequirements\nTo move your ML workflow to this stage, you’ll need to:\nMature streaming infrastructure with an\nefficient stream processing engine\nthat can compute all the streaming features with acceptable latency. You need to be able to get a request into your streaming transport and process them quickly enough for the prediction service before the next request arrives.\nA\nfeature store\nfor managing materialized features and ensuring consistency of stream features during training and prediction. Note:\ncurrent\nfeature stores\noften manage materialized streaming features but don’t manage feature computation or the source code for the features\n.\nA\nmodel store\n. A stream feature, after being created, needs to be validated. To ensure that a new feature actually helps with your model’s performance, you want to add it to a model, which efficiently creates a new model. Ideally, your model store should help you manage and evaluate models created with new streaming features, but model stores that also evaluate models don’t exist yet. You can potentially delegate part of this to a feature store.\nPreferably a better development environment\n. Data scientists currently work off historical data even when they’re creating streaming features, which makes it difficult to come up with and validate new streaming features. What if we can give data scientists direct access to data streams so that they can quickly experiment and validate new stream features? Instead of data scientists only having access to historical data, what if they can also access incoming streams of data from their notebooks?\nDiscussion: Online prediction for bandits and contextual bandits\nOnline prediction not only allows your models to make more accurate predictions but also enables bandits for online model evaluation – which is more interesting and more powerful than A/B testing – and as an exploration strategy for generating predictions.\nFor those unfamiliar, bandit algorithms originated in gambling. A casino has multiple slot machines with different payouts. A slot machine is also known as a one-armed bandit, hence the name. You don’t know which slot machine gives the highest payout. You can experiment over time to find out which slot machine is the best while maximizing your payout.\nMulti-armed bandits are algorithms that allow you to balance between exploitation (choosing the slot machine that has paid the most in the past) and exploration (choosing other slot machines that may pay off even more).\nBandits for model evaluation\nCurrently, the industry’s standard for online model evaluation is A/B testing. With A/B testing, you randomly route traffic to each model for predictions and measure at the end of your trial which model works better.\nA/B testing is stateless: you can route traffic to each model without having to know about their current performance. You can do A/B testing even with batch prediction.\nWhen you have multiple models to evaluate, each model can be considered a slot machine whose payout (e.g. prediction accuracy) you don’t know. Bandits allow you to determine how to route traffic to each model for prediction to determine the best model while minimizing wrong predictions shown to your users.\nBandits are stateful: before routing a request to a model, you need to calculate all models’ current performance.\nBandits are well-studied in academia and have been shown to be a lot more data-efficient than A/B testing. In many cases, bandits are even the optimal methods.\nBandits require less data to determine which model is the best, and at the same time, reduce opportunity cost as they route traffic to the better model more quickly.\nIn\nthis experiment\nby Google’s Greg Rafferty,\nA/B test required over 630,000 samples to get a confidence interval of 95%, while a simple bandit algorithm (Thompson Sampling) determined that a model was 5% better than the other with less than 12,000 samples.\nSee discussions on bandits at\nLinkedIn, Netflix, Facebook, Dropbox\n, and\nStitch Fix\n. For a more theoretical view, see chapter 2 of the\nReinforcement Learning book\n(Sutton & Barton, 2020).\nRequirements\nTo do bandits for model evaluation, your system needs the following three requirements.\nOnline prediction\n.\nPreferably short feedback loops\n: you need to get feedback on whether a prediction made by a model is good or not to calculate the models’ current performance. The feedback is used to extracted labels for predictions.\nExamples of tasks with short feedback loops tasks where labels can be determined from users’ feedback like in recommendations — if users click on a recommendation, the recommendation is inferred to be good. If the feedback loops are short, you can update the performance of each model quickly. If the loops are long, it’s still possible to do bandits, but it’ll take longer to update a model’s performance after it’s made a recommendation.\nA mechanism to collect feedback, calculate and keep track of each model’s performance, as well as route prediction requests to different models based on their current performance.\nBecause of these requirements, bandits are a lot more difficult to implement than A/B testing. Therefore, not widely used in the industry other than at a few big tech companies.\nContextual bandits as an exploration strategy\nIf bandits for model evaluation are to determine the payout (e.g. prediction accuracy) of each model, contextual bandits are to determine the payout of each action. In the case of recommendations, an action is an item to show to users, and the payout is how likely a user will click on it.\nDisclaimer\n:\nsome people also call bandits for model evaluation “contextual bandits”. This makes conversations confusing, so in this post, contextual bandits refer to exploration strategies to determine the payout of predictions.\nTo illustrate this, consider a recommendation system for 10,000 items. Each time, you can recommend 10 items to users. The 10 shown items get users’ feedback on them (click or not click). But you won’t get feedback on the other 9,990 items.\nIf you keep showing users only the items they most likely click on, you’ll get stuck in a feedback loop, showing only popular items and will never get feedback on less popular items.\nContextual bandits leverage contextual information to balance between showing users the items they will like (exploitation) and showing the items you don’t know much about yet (exploration) while minizing the cost of suboptimal actions.\nContextual bandits are well-researched and have been shown to improve models’ performance significantly (see reports by\nTwitter\n,\nGoogle\n). However, contextual bandits are even harder to implement than model bandits, since the exploration strategy depends on the ML model’s architecture (e.g. whether it’s a decision tree or a neural network), which makes it less generalizable across use cases.\nTowards Continual Learning\nWhen hearing continual learning, people imagine updating models very frequently, such as every 5 minutes. Many people argue that most companies don’t need updates that frequently because:\nThey don’t have traffic for that retraining schedule to make sense.\nTheir models don’t decay that fast.\nI agree with them. However, continual learning isn’t about the retraining frequency, but the manner in which the model is retrained.\nMost companies do\nstateless retraining\n– the model is trained from scratch each time. Continual learning means allowing\nstateful training\n– the model continues training on new data (fine-tuning).\nOnce your infrastructure is set up to do stateful training, the training frequency is just a knob to twist.\nYou can update your models once an hour, once a day, or you can update your models whenever your system detects a distribution shift.\nThere are two types of model updates:\nModel iteration\n: adding a new feature to an existing model architecture or changing the model architecture.\nData iteration\n: same model architecture and features but new data.\nAs of today, stateful training refers to data iteration. If you change your model architecture or add a new feature, you still have to train the new model from scratch. There has been interesting research showing that it’s possible to do\nknowledge transfer\n(Google, 2015) and\nmodel surgery\n(OpenAI, 2019): “\ntransfer trained weights from one network to another after a selection process to determine which sections of the model are unchanged and which must be re-initialized\n.” Several large research labs have experimented with this, though I’m not aware of any clear results in the industry.\nStarting with manual retraining, the first step will be to automate the retraining process. We’ll then move from stateless retraining to stateful training. Last but not least, we’ll discuss continual learning.\nStage 1. Manual, Stateless Retraining\nIn the beginning, your ML team focuses on developing ML models to solve as many business problems as possible – fraud detection, recommendation, delivery estimation, etc. Because your team is focusing on developing new models, updating existing models takes a backseat. You update an existing model only when:\nthe model’s performance has degraded to the point that it’s doing more harm than good, and\nyour team has time to update it.\nSome of your models are being updated once every six months. Some are being updated once a quarter. Some have been out in the wild for a year and haven’t been updated at all.\nThe process of updating a model is manual and ad-hoc. Someone, usually on the data platform team, queries data warehouses for new data. Someone else cleans this new data, extracts features from it, retrains that model from scratch on both the old and new data, then exports the updated model into a binary format. Then someone else takes that binary format and deploys the updated model. Oftentimes, the feature/model/processing code was updated during the model retraining, but the changes failed to be replicated to production, causing bugs that are hard to track down.\nIf this process sounds painfully familiar to you, you’re not alone. A vast majority of companies outside the tech industry – e.g. any company that has adopted ML less than 3 years ago and doesn’t have an ML platform team – are in this stage.\nStage 2. Automated Retraining\nInstead of retraining your model manually in an ad-hoc manner, you have a script to automatically execute the retraining process. This is usually done in a batch process, such as Spark.\nMost companies with somewhat mature ML infrastructure are in this stage. Some sophisticated companies run experiments to determine the optimal retraining frequency. However, for most companies, the retraining frequency is set based on gut feeling – e.g. “\nonce a day seems about right\n” or “\nlet’s kick off the retraining process each night when we have idle compute\n”.\nDifferent models in your pipeline might require different retraining schedules.\nIn the case of the session-based recommendation above, if your embedding model is separate from your ranking model, then the embedding model might need to be retrained a lot less frequently than the ranking model. For example, you might get away with retraining your embeddings once a week while retraining your ranking models once a day. It’s a different story if you have a lot of new items each day whose embeddings need to be learned.\nThings get a lot more complicated if there are dependencies among your models. For example, because the ranking model depends on the embeddings when the embeddings change, the ranking model should be updated too.\nRequirements\nIf your company has ML models in production, it’s likely that your company already has most of the infrastructure pieces needed for automated retraining. The only new piece you’ll need, if you don’t have one already, is\na model store\nto automatically version and store all the code/artifacts needed to reproduce a model.\nThe simplest model store is probably an S3 bucket that stores serialized blobs of models in some structured manner. If you want a more mature model store, the two solutions that I often hear about are SageMaker (managed service) and MLFlow (open-source). SageMaker is harder to use and doesn’t store your models’ code and artifacts. MLFlow is open-sourced and has more features but if your ML platform has a lot of quirks, it might be hard to get it to work.\nYou’ll need to write scripts to automate your workflow and configure your infrastructure to automatically sample your data, extract features, and process/annotate labels for retraining. How long this process will take depends on many factors, but here are the two major ones:\nScheduler\n. If you already have a CRON scheduler such as Airflow, Argo, wiring the scripts together shouldn’t be that hard.\nData access and availability\n. Is all the data you need already collected in your data warehouse? Will you have to join data from multiple organizations? Do you need to build a lot of tables from scratch? Stefan Krawczyk, ML/Data platform manager at Stitch Fix, commented that he suspects most people’s time might be spent here.\nBonus: Log and wait (feature reuse)\nWhen you retrain your model on new data, it’s likely that the new data has already gone through your prediction service, which means features have already been extracted once for predictions. Some companies reuse these extracted features for model updates, which both saves computation and allows for consistency between prediction and training. This approach is known as\nlog and wait\n. It’s a classic approach to reduce the training-serving skew – bugs caused by mismatches between the production and development environments.\nThis isn’t yet a popular approach, but it’s getting more popular. I expect that it’ll become a standard soon. Faire has a\ngreat blog post\ndiscussing the pros and cons of their\nlog and wait\napproach.\nStage 3. Automated, Stateful Training\nRemember that stateful training is when you continue training your model on new data instead of retraining your model from scratch.\nStateful retraining allows you to update your model with less data.\nIf you update your model once a month, you might need to retrain your model from scratch on data from the last 3 months. However, if you update your model every day, you only need to fine-tune your model on data from the last day.\nGrubhub, after switching from stateless daily retraining to stateful daily retraining, reduced their training cost\n45 times\n(2021).\nStateless vs. stateful training\nAnother nice property of incremental learning is that you only need to see each data sample at most twice: once during prediction and once during model training. If you’re worried about data privacy, you might be able to discard your data samples after using them.\nRequirements\nThe main thing you need at this stage is a better model store.\nModel lineage\n: you want to not just version your models but also track their lineage – which model fine-tunes on which model.\nStreaming features reproducibility\n: you want to be able to time-travel to extract streaming features in the past and recreate the training data for your models at any point in the past in case something happens and you need to debug.\nAs far as I know, no existing model store has both of these capacities. You might be able to delegate streaming features reproducibility to a feature store, but you’ll likely have to build the solution in-house.\nStage 4. Continual Learning\nWhat I’m working towards and what I hope many companies will eventually adopt is continual learning.\nInstead of updating your models based on a fixed schedule, continually update your model whenever data distributions shift and the model’s performance plummets.\nThe holy grail is when you combine continual learning with edge deployment. Imagine you can ship a base model with a new device – a phone, a watch, a drone, etc. — and the model on that device will continually update and adapt to its environment. There’s no centralized server cost, no need to transfer data back and forth between device and cloud!\nRequirements\nThe switch from stage 3 to stage 4 is steep. You’ll need the following:\nA mechanism to trigger model updates\n. This trigger can be time-based (e.g. every 5 minutes), performance-based (e.g. whenever a model performance plummets), or drift-based (e.g. whenever data distributions shift).\nMost monitoring solutions today focus on analyzing features – analyzing the summary statistics of a feature (e.g. mean, variance, min, max) and alerting you when significant changes in these statistics happen. However, a model can have hundreds, if not thousands of features. Most feature statistics changes are benign. The problem is not how to detect these changes, but how to know which change actually requires your attention.\nBetter ways to continually evaluate your models\n. Writing a function to update your models isn’t much different from what you’d do in stage 3. The hard part is to ensure that the updated model is working properly. Because you’re updating your models to adapt to changing environments, a stationary test set no longer suffices. You might want to incorporate backtest,\nprogressive evaluation\n, test in production including shadow deployment, A/B test, canary analysis, and bandits.\nAn orchestrator\nto automatically spin up instances to update and evaluate your models without interrupting the existing prediction service.\nConclusion\nReal-time machine learning is largely an infrastructure problem. Solving it will require the data science/ML team and the platform team to work together.\nBoth online inference and continual learning require a mature streaming infrastructure. The training part of continual learning can be done in batch, but the online evaluation part requires streaming. Many engineers worry that streaming is hard and costly. It was true 3 years ago, but streaming technologies have matured significantly since then. More and more companies are providing solutions to make it easier for companies to move to streaming, including Spark Streaming, Snowflake Streaming, Materialize, Decodable, Vectorize, etc.\nTo better understand the adoption and challenges of real-time ML in the industry, my team and I are doing a\nsurvey\n. It’d be great if you could share with us your thoughts –  it should take around 5 minutes. The results will be aggregated, summarized, and shared with the community. Thank you!\nDo get in touch if you want to discuss how I and my team can help you with online prediction, online model evaluation, and automated, stateful training.\nAcknowledgments\nThis post wouldn’t have been possible without the help of a long list of engineers who have been extremely patient with my questions about their workflows.\nI’d also like to thank\nStefan Krawczyk\n,\nZhenzhong Xu\n,\nMax Halford\n,\nGoku Mohandas\n,\nAlex Egg\n,\nJacopo Tagliabue\n, and\nLuke Metz\nfor their thoughtful comments that made this post better.\nThanks Ammar Asmro, Alex Gidiotis, Aditya Soni, Shaosu Liu, Deepanshu Setia, Bonson Sebastian Mampilli, Sean Sheng, and Daniel Tan for your feedback on the survey!\nAppendix\nOn the efficiency of stream processing vs. batch processing\nWe’ll compare the efficiency along three dimensions: overall cost efficiency, compute performance efficiency, and talent efficiency.\nThanks Zhenzhong Xu for this thoughtful analysis!\nCost efficiency\nUnfortunately, there is no universal cost-efficiency formula between stream and batch processing. It depends on a lot of variables such as latency requirement, data size, late arrival tolerance, failure tolerance, statefulness, etc.  Stream processing’s strength is in stateful, continuous, and unbounded data processing. When used properly, often than not it’s less expensive compared to stateless batch processing. An example is if you have to compute a sliding window to score the user engagement during their 30 day trial period, batch processing will have to re-compute 30 days’ worth of data in every single batch, while stateful streaming workload will avoid all the redundant processing, hence less computationally expensive. On the other hand, for one-time processing of extremely large datasets at rest, Batch will still be the top choice. The best teams are the ones who master both.\nPerformance efficiency\nStream Processing is a very sophisticated technology today. Frameworks like Apache Flink are proven to be highly scalable and fully distributed. Stream Processing is optimized for speed and unbounded data. Its performance is measured by throughput per operator, and the availability SLA is often defined as the percentage of events processed within a certain delay tolerance. However, Stream Processing is not optimized for large bounded data processing, and its performance is significantly lower in large historical data processing (e.g., backfill). With that being said, having a cohesive and unified architecture leveraging both streaming and batch’s strength is an active development area. The takeaway here is stream processing and batch all have their strengths and weaknesses. But ultimately as a data processing user, you probably don’t need to worry about the current streaming/batch divide, the gap will close in a couple of years as the abstraction inevitably moves up.\nTalent efficiency\nToday, it’s still a general consensus that streaming is hard to manage in-house. You will need a very strong team of infrastructure engineers who are well versed to do both technical depth and operation work.  However, with data processing capabilities becoming more and more commoditized, many companies will start considering buying rather than building in-house. An analogy here, you don’t need nuclear scientists to work for you if what you need is to turn electricity into higher values. (oh well, unless you are running a nuclear powered electricity company  :) - in that case, be ready to get the best people you can!)\nWhat do feature stores do?\nThanks Stefan Krawczyk for this analysis.\nFeature stores like Tecton and FEAST do some streaming feature computation for you, but they only operate on data that doesn’t need any joins, so it’s unlikely they’ll orchestrate an entire feature computation flow for you.\nSagemaker’s feature store\nonly stores materialized data and requires you to hook it up to pipelines that would do the actual computation. People then reference/use the materialized features in the system, not the actual code that computes them.\nIf you want to learn any of the above topics, join our\nDiscord chat\n. We’ll be sharing learning resources and strategies. We might even host learning sessions and discussions if there’s interest. Serious learners only!",
  "metadata": {
    "title": "Real-time machine learning: challenges and solutions",
    "description": "[Twitter discussion, LinkedIn]",
    "keywords": null,
    "author": "Chip Huyen",
    "og:title": "Real-time machine learning: challenges and solutions",
    "og:locale": "en_US",
    "og:description": "[Twitter discussion, LinkedIn]",
    "og:url": "https://huyenchip.com/2022/01/02/real-time-machine-learning-challenges-and-solutions.html",
    "og:site_name": "Chip Huyen",
    "og:image": "https://huyenchip.com/assets/pics/real-time-ml-2022/online-prediction.png",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "article:published_time": "2022-01-02T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/real-time-ml-2022/batch-prediction.png",
      "bytes": 203490,
      "reused": false,
      "local_path": "attachments/e61555102aed/images/body-8c1ec78b7f17774beda9.png",
      "src": "https://huyenchip.com/assets/pics/real-time-ml-2022/batch-prediction.png",
      "alt": "A typical batch prediction workflow",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/real-time-ml-2022/session-recsys.png",
      "bytes": 209105,
      "reused": false,
      "local_path": "attachments/e61555102aed/images/body-3aa3299bca6b0aaca004.png",
      "src": "https://huyenchip.com/assets/pics/real-time-ml-2022/session-recsys.png",
      "alt": "A typical batch prediction workflow",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/real-time-ml-2022/online-prediction.png",
      "bytes": 461112,
      "reused": false,
      "local_path": "attachments/e61555102aed/images/body-64c32fe3fc9b21cf9d0b.png",
      "src": "https://huyenchip.com/assets/pics/real-time-ml-2022/online-prediction.png",
      "alt": "A typical batch prediction workflow",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/real-time-ml-2022/stateful-training.png",
      "bytes": 375736,
      "reused": false,
      "local_path": "attachments/e61555102aed/images/body-63010962d598be5c7eee.png",
      "src": "https://huyenchip.com/assets/pics/real-time-ml-2022/stateful-training.png",
      "alt": "Stateless vs. stateful training",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://twitter.com/chipro/status/1477812589484658689",
      "label": "Twitter discussion",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/posts/chiphuyen_machinelearning-realtimeml-onlineprediction-activity-6883579455357489152-u1I0",
      "label": "LinkedIn",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969",
      "label": "Designing Machine Learning Systems",
      "type": "a"
    },
    {
      "href": "https://huyenchip.com/2020/12/27/real-time-machine-learning.html",
      "label": "machine learning is going real-time",
      "type": "a"
    },
    {
      "href": "https://forms.gle/dDvgF7QgpPdvJE5b8",
      "label": "5-minute survey",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/real-time-recommendations/",
      "label": "session-based recommendations",
      "type": "a"
    },
    {
      "href": "https://developers.google.com/machine-learning/recommendation/overview/candidate-generation",
      "label": "Google’s free crash course",
      "type": "a"
    },
    {
      "href": "https://www.businessofapps.com/data/grubhub-statistics/",
      "label": "31 million users",
      "type": "a"
    },
    {
      "href": "https://www.statista.com/statistics/668094/orders-per-day-grubhub/",
      "label": "622,000 daily orders",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.12133",
      "label": "SQL-like joins",
      "type": "a"
    },
    {
      "href": "https://madewithml.com/courses/mlops/feature-store/",
      "label": "feature stores",
      "type": "a"
    },
    {
      "href": "https://towardsdatascience.com/a-b-testing-is-there-a-better-way-an-exploration-of-multi-armed-bandits-98ca927b357d",
      "label": "this experiment",
      "type": "a"
    },
    {
      "href": "https://netflixtechblog.com/ml-platform-meetup-infra-for-contextual-bandits-and-reinforcement-learning-4a90305948ef",
      "label": "LinkedIn, Netflix, Facebook, Dropbox",
      "type": "a"
    },
    {
      "href": "https://multithreaded.stitchfix.com/blog/2020/08/05/bandits/",
      "label": "Stitch Fix",
      "type": "a"
    },
    {
      "href": "http://incompleteideas.net/book/RLbook2020.pdf",
      "label": "Reinforcement Learning book",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2008.00727",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.03212",
      "label": "Google",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1511.05641",
      "label": "knowledge transfer",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.06719",
      "label": "model surgery",
      "type": "a"
    },
    {
      "href": "https://craft.faire.com/building-faires-new-marketplace-ranking-infrastructure-a53bf938aba0",
      "label": "great blog post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2107.07106",
      "label": "45 times",
      "type": "a"
    },
    {
      "href": "https://maxhalford.github.io/blog/online-learning-evaluation/",
      "label": "progressive evaluation",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/skrawczyk/",
      "label": "Stefan Krawczyk",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/zhenzhong-xu-0243003/",
      "label": "Zhenzhong Xu",
      "type": "a"
    },
    {
      "href": "https://maxhalford.github.io/",
      "label": "Max Halford",
      "type": "a"
    },
    {
      "href": "https://twitter.com/gokumohandas",
      "label": "Goku Mohandas",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/eggie5/",
      "label": "Alex Egg",
      "type": "a"
    },
    {
      "href": "https://twitter.com/jacopotagliabue",
      "label": "Jacopo Tagliabue",
      "type": "a"
    },
    {
      "href": "http://lukemetz.com/",
      "label": "Luke Metz",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/sagemaker/feature-store/",
      "label": "Sagemaker’s feature store",
      "type": "a"
    },
    {
      "href": "https://discord.gg/Mw77HPrgjF",
      "label": "Discord chat",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:51Z",
  "source_url": "https://huyenchip.com/2022/01/02/real-time-machine-learning-challenges-and-solutions.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.12133",
      "bytes": 714951,
      "reused": false,
      "local_path": "papers/repair/38bf0a310d229d18fe7b.pdf"
    },
    {
      "status": "ok",
      "url": "http://incompleteideas.net/book/RLbook2020.pdf",
      "bytes": 73129769,
      "reused": false,
      "local_path": "papers/repair/cad5b62f430ff6b0eed4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2008.00727",
      "bytes": 1093231,
      "reused": false,
      "local_path": "papers/repair/7fe14ea8ba39c924b2a5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.03212",
      "bytes": 887973,
      "reused": false,
      "local_path": "papers/repair/83ac6b35f65ed2b14102.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.05641",
      "bytes": 514031,
      "reused": true,
      "local_path": "papers/repair/92e6dc487fb58180dcc6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.06719",
      "bytes": 426876,
      "reused": false,
      "local_path": "papers/repair/3b44759e96435ba556f4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2107.07106",
      "bytes": 629020,
      "reused": false,
      "local_path": "papers/repair/1dd02881423d73031668.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 35550,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e61555102aed/pages/source.html",
  "local_html": "attachments/e61555102aed/pages/article.html",
  "body_base_path": "attachments/e61555102aed/pages/",
  "body_markdown_path": "attachments/e61555102aed/pages/article.md"
}