{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Chip Huyen",
  "title_listed": "Introduction to streaming for data scientists",
  "keywords": "流处理,Kafka,数据科学",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huyenchip.com/2022/08/03/stream-processing-for-data-scientists.html",
  "publish_date": "2022-08",
  "corpus_stage": "数据处理",
  "notes": "面向数据科学家的流处理入门",
  "id": "80b0bf3d0f72",
  "collected_at": "2026-10-02T14:14:20",
  "title": "Introduction to streaming for data scientists",
  "body_markdown": "As\nmachine learning moves towards real-time\n, streaming technology is becoming increasingly important for data scientists. Like many people coming from a machine learning background, I used to dread streaming. In our recent survey, almost half of the data scientists we asked said they would like to move from batch prediction to online prediction but can’t because streaming is hard, both technically and operationally. Phrases that the streaming community take for granted like “time-variant results”, “time travel”, “materialized view” certainly don’t help.\nOver the last year, working with\na co-founder who’s super deep into streaming\n, I’ve learned that streaming can be quite intuitive. This post is an attempt to rephrase what I’ve learned.\nWith luck, as a data scientist, you shouldn’t have to build or maintain a streaming system yourself. Your company should have infrastructure to help you with this. However, understanding where streaming is useful and why streaming is hard could help you evaluate the right tools and allocate sufficient resources for your needs.\nQuick recap: historical data vs. streaming data\nOnce your data is stored in files, data lakes, or data warehouses, it becomes historical data.\nStreaming data refers to data that is still flowing through a system, e.g. moving from one microservice to another.\nBatch processing vs. stream processing\nHistorical data is often processed in batch jobs — jobs that are kicked off periodically. For example, once a day, you might want to kick off a batch job to generate recommendations for all users. When data is processed in batch jobs, we refer to it as batch processing. Batch processing has been a research subject for many decades, and companies have come up with distributed systems like MapReduce and Spark to process batch data efficiently.\nStream processing refers to doing computation on streaming data. Stream processing is relatively new. We’ll discuss it in this post.\n1. Where is streaming useful in machine learning?\nThere are three places in the ML production pipeline where streaming can matter a lot: online prediction, real-time observability, and continual learning.\nOnline prediction (vs. batch prediction)\nBatch prediction means periodically generating predictions offline, before prediction requests arise. For example, every 4 hours, Netflix might generate movie recommendations for all of its users. The recommendations are stored and shown to users whenever they visit the website.\nOnline prediction means generating predictions on-demand, after prediction requests arise. For example, an ecommerce site might generate product recommendations for a user whenever this user visits the site. Because recommendations are generated online, latency is crucial.\nImagine you work for an ecommerce site and want to predict the optimal discount to give a user for a given product to encourage them to buy it. One of the features you might use is the average price of all the items this user has looked at in the last 30 minutes. This is an\nonline feature\n– it needs to be computed on online data (as opposed to being pre-computed on historical data).\nOne “easy” way to do this is to spin up a service like AWS Lambda and Google Cloud Function to compute this feature whenever requests arrive. This is how many companies are doing it and it works in many simple cases. However, it doesn’t scale if you want to use complex online features.\nThis “easy” way is stateless, which means that each computation is independent from the previous one. Every time you compute this feature, you have to process all the data from the last 30 minutes, e.g:\nTo compute this feature at 10:33am, you’ll have to process data from 10:03 to 10:33.\nTo compute this feature again at 10:35, you’ll have to process data from 10:05 to 10:35.\nThe data from 10:05 to 10:33 is processed twice!\nStream processing can be stateless too, but stateless stream processing is kinda boring, so we’ll only talk about stateful stream processing here. Stateful stream processing can avoid redundancy, hence faster and cheaper. For example, when computing this feature again at 10:35, you’ll only need to process data from 10:33 to 10:35 and join the new result with the previous result.\nReal-time monitoring (vs. batch monitoring)\nThere’s a world of difference between batch monitoring and real-time monitoring, and many monitoring solutions (especially batch solutions) conflate the two.\nBatch monitoring means running a script to compute the metrics you care about periodically (like once a day), usually on data in warehouses like S3, BigQuery, Snowflake, etc. Batch monitoring is slow. You first have to wait for data to arrive at warehouses, then wait for the script to run. If your new model fails upon deployment, you won’t find out at least until the next batch of metrics is computed, assuming the metrics computed are sufficient to uncover the issue. And you can’t fix the failure before you find out about it.\nReal-time monitoring means computing metrics on data as it arrives, allowing you to get insights into your systems in (near) real-time. Like online feature computation, real-time metric computation can often be done more efficiently with streaming.\nIf batch monitoring works for you, that’s great. In most use cases, however, batch monitoring often ends up being not enough. In software engineering, we know that the iteration speed is everything – cue\nBoyd’s Law of Iteration: speed of iteration beats quality of iteration\n. The DevOps community has been obsessed with\nspeeding up iterations for higher quality\nsoftware for over two decades. Two important metrics in DevOps are:\nMTTD\n: mean time to detection\nMTTR\n: mean time to response\nIf something is going to fail, you want it to fail fast, and you want to know about it fast, especially before your customers do. If your user has to call you in the middle of the night complaining about a failure or worse, tweet about it, you’re likely to lose more than one customer.\nYet, most people doing MLOps are still okay with day-long, even week-long feedback loops.\nMany teams, without any monitoring system in place for ML, want to start with batch monitoring. Batch monitoring is still better than no monitoring, and arguably easier to set up than real-time monitoring.\nHowever, batch monitoring isn’t a prerequisite for real-time monitoring – setting up batch monitoring today is unlikely to make it easier to set up real-time monitoring later. Working on batch monitoring first means another legacy system to tear down later when you need to get real-time.\nContinual learning (vs. manual retraining)\nContinual learning\nrefers to the ability to update your models whenever needed and to deploy this update quickly. Fast model updates enable our models to adapt to changing environments and business requirements, especially to\ndata distribution shifts\n.\nI’ve talked to ML engineers / data scientists at over 100 companies. Almost everyone told me that continual learning is too “far out”. Most companies are still struggling with dependency management and monitoring to think about continual learning.\nAt the same time, nobody has ever told me that they don’t want continual learning. Continual learning is too “far out” because we don’t yet have the right tool for it. When the right tool emerges that makes continual learning “just work”, I believe the transition will be swift.\nWe don’t need streaming to continually update our models\n. Assuming that you already have a way of accessing fresh data, the hard part of continual learning isn’t in updating the models, but in ensuring that the updated models work. We need to find a way to evaluate each model iteration fast, which brings us back to real-time monitoring and streaming.\n2. From table to log\nI hope that I’ve convinced you why and where we need streaming in ML. If you’re not convinced, I’d love to hear from you. Now, let’s walk through some of the core concepts behind streaming that I find really cool.\nThe first article that helped me make sense of streaming is the classic post\nThe log\nby Jay Kreps, a creator of Kafka.\nSide note\n: Kreps mentioned in a\ntweet\nthat he wrote the post to see if there was enough interest in streaming for his team to start a company around it. The post must have been popular because his team spun out of LinkedIn to become Confluent.\nThe core idea of the post (for me) is the duality of\ntable\nand\nlog\n. You’re probably already familiar with data\ntables\n, like a MySQL table or a pandas DataFrame. For simplicity, consider the following inventory table.\nBefore 2022-06-19, the price of item #3 is $30\nLet’s say, on June 19, you want to make a change to this table – e.g. changing the price of item #3 (Blouse) from $30 to $35. There are many ways to do this, but the 2 common approaches are:\nGo to where the table is stored and make the change directly (e.g. overwriting a file). This means that we’ll have no way of reverting the change, but many companies do it anyway.\nSend an update with the new price\n{\"timestamp\": \"2022-06-19 09:23:23\", \"id\": 3, \"Price ($)\": 35}\n, and this update will be applied to the table. Overtime, we’ll have a series of ordered updates, which is called a\nlog\n. Each update is an example of an\nevent\n. Logs are append-only. You can only append the new events to your existing log. You can’t overwrite previous events.\n…\n{\"timestamp\": \"2022-06-08 11:05:04\", \"id\": 3, \"Item\": \"Blouse\", \"Price ($)\": 30}\n{\"timestamp\": \"2022-06-17 03:12:16\", \"id\": 4, \"Item\": \"Jeans\", \"Price ($)\": 40}\n{\"timestamp\": \"2022-06-19 09:23:23\", \"id\": 3, \"Item\": \"Blouse\", \"Price ($)\": 35}\nI’m using wall clock timestamps for readability. In practice, wall clocks are unreliable. Distributed systems often leverage\nlogical clock\ninstead.\nThe data table after the price of item #3 has been updated to $35\nA table captures the state of data at a point in time. Given a table alone, we don’t know what the state of data was a day ago or a week ago. Given a log of all the changes to this table, we can recreate this table at any point in time.\nYou’re already familiar with logs if you work with git. A git log keeps track of all the changes to your code. Given this log, you can recreate your code at any point in time.\nStateless vs. stateful log\nIt’s possible to send an event with the price difference only\n{\"id\": 3, \"Action\": \"increase\", \"Price ($)\": 5}\n. For this type of event we’ll need to look at the previous price to determine the updated price.\nThis is called a stateful event (we need to know the previous state to construct the current state). The type of event mentioned in #2 is a stateless log. Stateful events are required to be processed in order – adding $5 to the blouse price then doubling it is very different from doubling it then adding $5.\nThere’s a whole industry focusing on solving stateful events called\nChange Data Capture\n(CDC). See\nDebezium\n,\nFivetran\n,\nStriim\n.\nReal-time transport and computation engine\nThere are two components of a streaming system: the real-time transport and the computation engine.\nThe\nreal-time transport\n, which are basically distributed logs. Data in real-time transports are called streaming data, or “data in motion” as coined by Confluent. Examples: Kafka, AWS Kinesis, GCP Dataflow.\nThe\ncomputation engine\nperforms computation (e.g. joining, aggregation, filtering, etc.) on the data being transported. Examples: Flink, KSQL, Beam, Materialize, Decodable, Spark Streaming.\nTransports usually have capacity for simple computation. For example, Kafka is a transport that also has an API called Kafka Streams that provides basic stream processing capacity. However, if you want to perform complex computation, you might want an optimized stream computation engine (similar to how bars usually have some food options but if you want real food you’ll want to go to a restaurant).\nWe’ve discussed logs at length. From the next section onward, we’ll focus on stream computation.\n3. A simple example of stream processing\nImagine that we want to use the average price of all items a user has looked at as a feature. If we put all these items into a table, the feature definition might look something like this. This is batch processing.\n# Pseudocode\ndef average_price(table):\n    return sum(table.price) / len(table)\nIf, instead, we have a log of all item updates, how should we compute the average price?\nInternal state and checkpoints\nOne way to do it is to process events in the log (replay the log) from the beginning of time to recreate the inventory table of all items. However, we actually don’t need to keep track of the whole table to get the average price – we just need the total price of all items and the item count. The average price can be computed from these two values.\nThe total price and the count constitute the\ninternal state\nof the stream processing job. We say this computation is stateful because we keep track of the internal state between computation.\n# Pseudocode\ndef average_price(log):\n    initialize total_price, item_count\n    for each event in the log:\n        update total_price, item_count\n    avg_price = total_price / item_count\nreturn avg_price\nIf the log is large (in 2021,\nNetflix was processing 20 trillion events a day!\n), replaying the log from the beginning can be prohibitively slow. To mitigate this, we can occasionally save the internal state of the job, e.g. once every hour. The saved internal state is called a\ncheckpoint\n(or savepoint), and this job can resume from any checkpoint. The function above will be updated as follows:\n# Pseudocode\ndef average_price(log):\n    find the latest checkpoint X\n    load total_price, item_count from X\n    for each event in the log after X:\nupdate total_price, item_count\navg_price = total_price / item_count\nreturn avg_price\nMaterialized view\nThe computed average price value, if saved, will become a\nmaterialized view\nof the feature average price. Each time we query from the materialized view, we don’t recompute the value, but read the saved value.\nYou might wonder: “Why must we come up with a new term like\nmaterialized view\ninstead of saying\nsaved copy\nor\ncache\n?”\nBecause “saved copy” or “cache” implies that the computed value will never change. Materialized views can change due to late arrivals. If an event that happened at 10.15am got delayed and arrived at 11.10am, all the materialized views that might be affected by this update\nshould\nbe recomputed to account for this delayed event. I say “should” because not all implementations do this.\nReading from a materialized view is great for latency, since we don’t need to recompute the feature. However, the materialized view will eventually become outdated. Materialized views will need to be\nrefreshed\n(recomputed) either periodically (every few minutes) or based on a change stream.\nWhen refreshing a materialized view, you can recompute it from scratch (e.g. using all the items to compute their average price). Or you can update it using only the new information (e.g. using the latest materialized average price + the prices of updated items). The latter is called\nincremental materialized\n. For those interested, Materialize has a great article with examples on\nmaterialized and incremental materialized\n.\n4. Time travel and backfilling\nWe’ve talked about how to compute a feature on the latest data. Next, we’ll discuss how to compute a feature in the past.\nImagine that the latest price for the item blouse in our inventory is $35. We want to compute the\navg_price\nfeature as it would’ve happened on June 10. There are two ways this feature might have differed in the past:\nThe data was different. For example, the blouse’s price was $30 instead of $35.\nThe feature logic was different. For example:\nToday, the\navg_price\nfeature computes the\naverage raw price\nof all items.\nOn June 10, the\navg_price\nfeature computes the\naverage listed price\n, and the listed price is the raw price plus discount.\nIf we accidentally use the data or the feature logic from today instead of the data & logic from June 10, it means there is a leakage from the future.\nPoint-in-time correctness\nrefers to a system’s ability to accurately perform a computation as it would’ve happened at any time in the past. Point-in-time correctness means no data leakage.\nRetroactively processing historical data using a different logic is also called\nbackfilling\n. Backfilling is a very common operation in data workflows, e.g. see backfilling in\nAirflow\nand\nAmplitude\n. Backfilling can be done both in batch and streaming.\nWith batch backfilling, you can apply the new logic (e.g. new feature definition) to a table in the past.\nWith stream backfilling, you can apply the new logic to the log in a given period of time in the past, e.g. apply it to the log on June 10, 2022.\nTime travel is hard, because we need to keep track of the states of your data over time. It can get very complicated if your data is joined from multiple places. For example, the average listed price depends on raw prices and discounts, both of them can independently change over time.\nTime travel in ML workflows\nThere are many scenarios in an ML workflow where time travel is necessary, but here are the two major ones:\nto ensure the train-predict consistency for online features (also known as online-offline skew or training-serving skew)\nto compare two models on historical data\nTrain-predict consistency for online features\nOnline prediction\ncan cause train-predict inconsistency. Imagine our model does online prediction using online features, one of them is the average price of all the items this user has looked at in the last 30 minutes. Whenever a prediction request arrives, we compute this feature using the prices for these items\nat that moment\n, possibly using stream processing.\nDuring training, we want to use this same feature, but computed on historical data, such as data from the last week. Over the last week, however, the prices for many items have changed, some have changed multiple times. We need time travel to ensure that the historical average prices can be computed correctly at each given point in time.\nProduction-first ML\nTraditionally, ML workflows are development-first\n. Data scientists experiment with new features for the training pipeline. Feature definitions, aka feature logic, are often written for batch processing, blissfully unaware of the time-dependent nature of data. Training data often doesn’t even have timestamps.\nThis approach might help data scientists understand how a model or feature does during training, but not how this model / feature will perform in production, both performance-wise (e.g. accuracy, F1) and operation-wise (e.g. latency).\nAt deployment, ML or ops engineers need to translate these batch features into streaming features for the prediction pipeline and optimize them for latency. Errors during this translation process – e.g. changes or assumptions in one pipeline not updated or insufficiently tested in another – create train - predict inconsistency.\nThis inconsistency gets worse when the training and prediction pipelines use different languages, e.g. the training pipeline uses Python and the prediction pipeline uses SQL or Java.\nTo avoid this inconsistency, we should get rid of the translation process altogether. What if we let data scientists experiment with features for the prediction pipeline directly? You can write new features as they will be used in production. To test out these new features, you apply them to historical data to generate training data to train models. Applying new feature logic to historical data is exactly what backfilling is. With this setup,\nML workflows become production-first\n!\nThanks to its ability to support both streaming and batch processing, SQL has emerged to be a popular language for production-first ML workflows. In the relational database era, SQL computation engines were entirely in the batch paradigm. In recent years, there has been a lot of investment in\nstreaming SQL engines\n, such as KSQL and Apache Flink. (See\nStreaming SQL to Unify Batch & Stream Processing w/ Apache Flink @Uber\n)\nDevelopment-first ML workflows: features are written for training and adapted for online prediction.\nProduction-first ML workflows: features are written for online prediction and backfilled for training.\nBacktest for model comparison\nBackfilling to generate features for training is how feature stores like FeatureForm and Tecton ensure train-predict consistency. However, most feature stores only support limited streaming computation engines, which can pose performance (latency + cost) challenges when you want to work with complex online features.\nExample of a feature definition in SQL [FeatureForm]\nExample of a feature definition in SQL [Tecton]\nImagine your team is using model A to serve live prediction requests, and you just came up with a new model B. Before A/B testing model B on live traffic, you want to evaluate model B on the most recent data, e.g. on the prediction requests from the last day.\nTo do so, you replay the requests from the last day, and use model B to generate predictions for those requests to see what the performance would have been if we had used B instead of A.\nIf model B uses exactly the same feature logic as model A, model B can reuse the features previously computed for model A. However, if model B uses a different feature set, you’ll have to backfill, apply model B’s feature logic to data from the last day, to generate features for model B.\nThis is also called a backtest.\nTime travel challenges for data scientists\nSome data science teams have told me that they need to do time travel for their data but their data doesn’t have timestamps. What should they do?\nIt’s impossible to time travel without timestamps. This is not a problem that data scientists can solve without having control over how data is ingested and stored.\nSo why don’t data scientists have access to data with timestamps?\nOne reason is that data platforms might have been designed using practices from an era where ML production wasn’t well understood. Therefore, they were not designed with ML production in mind, and timestamps weren’t considered an important artifact to keep track of.\nAnother reason is the lack of communication between those who ingest the data (e.g. the data platform team) and those who consume the data (e.g. the data science team). Many companies already have application logs, however, the timestamps are not retained during data ingestion or processing, or they are retained somewhere that data scientists don’t access to.\nThis is an operational issue that companies who want modern ML workflows will need to invest into. Without a well-designed data pipeline, data scientists won’t be able to build production-ready data applications.\nthe reason data quality is bad is because most data teams have zero power over the creation of data. it's like a soviet supply chain, where the end consumer has to accept whatever the factory churns out.\n— Slater Stich (@slaterstich)\nJune 1, 2022\n5. Why is streaming hard?\nI hope that by now, I’ve convinced you why streaming is important for ML workloads. Another question is: if streaming is so important, why is it not more common in ML workflows?\nBecause it’s hard.\nIf you’ve worked with streaming before, you probably don’t need any convincing. Tyler Aikidau gave a fantastic talk on the\nopen challenges in stream processing\n. Here, I’ll go over the key challenges from my perspective as a data scientist. This is far from an exhaustive list. A streaming system needs to be able to handle all of these challenges and more.\nDistributed computation\nStreaming is simple when you have a small amount of updates and they are generated and consumed by one single machine, like the inventory example in\nFrom table to log\n. However, a real-world application can have hundreds, if not thousands, of microservices, spread out over different machines. Many of these machines might try to read and write from the same log. Streaming systems are highly distributed. Engineers working\ndirectly\nwith streaming need to have a deep understanding of distributed systems, which not many data scientists would’ve exposure to in their day-to-day jobs. We need good abstraction to enable data scientists to leverage streaming without having to deal with the underlying distributed systems.\nTime-variant results\nTime adds another dimension of complexity. If you apply the same query to the same table twice, you’ll get the same result. However, if you apply the same query to the same stream twice, you’ll likely get very different results.\nIn batch, if a query fails, you can just rerun it. But in streaming, due to time-variant results, if a query fails, what should we do? Different stream processing engines have different ways to address this problem. For example, Flink uses the\nChandy–Lamport algorithm\n, developed by K. Mani Chandy and the Turing Award winner Leslie Lamport.\nCascading failure\nIn a batch setting, jobs are typically scheduled at large intervals, hence fluctuation in workloads isn’t a big deal. If you schedule a job to run every 24 hours and one run takes 6 hours instead of 4 hours, these extra 2 hours won’t affect the next run.\nHowever, fluctuation in a streaming setting can make it very hard to maintain consistently low latency. Assuming that your streaming system is capable of handling 100 events / second. If the traffic is less than 100 events / second, your system is fine. If the traffic suddenly surges to 200 events one second, ~100 events will be delayed. If not addressed quickly, these delayed 100 seconds will overload your system the next second, and so on, causing your system to significantly slow down.\nAvailability vs. consistency\nImagine that there are two machines trying to send updates about the same item:\nMachine A, at “03:12:10”, updates that user views product 123\nMachine B, at “03:12:16”, updates that user views product 234\nDue to some network delays, machine A’s update is delayed in transit, and arrives later than machine B’s update. How long should we wait for machine A before computing the average prices of all items the user has looked at in the last 30 minutes?\nIf we don’t wait long enough, we’ll miss machine A’s update, and the feature will be incorrect. However, if we wait too long, the latency will be low. An optimal time window is probably somewhere in the middle. Therefore, the results of stream computation tend to be\napproximate\n.\nThis leads us to an interesting trade-off:\navailability vs. consistency\n. For our system to be highly available, we’ll want to duplicate the same workload over multiple machines. However, the more machines there are, the more stragglers there will be, and the more inconsistent the results of your stream computation will become. All large-scale distributed systems, not just streaming systems, have to make this tradeoff.\nOperational challenges\nWhile streaming technology can be technically hard, the real challenge for companies to adapt streaming is in operations. Operating on (near) real-time data streams means that we can make things happen in (near) real-time. Unfortunately, this means that catastrophes can also happen in (near) real-time.\nMaintaining a streaming system means that your engineers will have to be on-call 24x7 to respond to incidents quickly. If you can’t figure out the problems and resolve them fast enough, data streams will be interrupted and you might lose data. Because the stake is higher in streaming systems, maintenance of streaming systems requires a high level of automation and mature DevOps practices.\nWe’ve talked to many small teams who want to maintain their own Kafka or Flink clusters and it’s not pretty. Streaming is an area where I can see a lot of values in managed services (e.g. Confluent instead of Kafka).\nConclusion\nWow, this was a hard post for me to write. I hope you’re still with me.\nI’m very excited about the application of streaming for ML applications. My dream is that data scientists should have access to an ML platform that enables them to leverage streaming without having to worry about its underlying complexities. This platform needs to provide a high-enough level of abstraction for things to just work, while being flexible enough for data scientists to customize their workflows.\nI know that leveraging streaming for ML is hard right now and I believe in the value of making it easier, so I’m working on it. Hit me up if you want to chat!\nThis is a relatively new field, and I’m still learning too. I’d appreciate any feedback and discussion points you have!\nAcknowledgment\nThanks Zhenzhong Xu for having patiently answered all my dumb questions on streaming,\nLuke Metz\n,\nChloe He\n,\nHan-chung Lee\n,\nDan Turkel\n, and\nRobert Metzger\n.",
  "metadata": {
    "title": "Introduction to streaming for data scientists",
    "description": "As machine learning moves towards real-time, streaming technology is becoming increasingly important for data scientists. Like many people coming from a mach...",
    "keywords": null,
    "author": "Chip Huyen",
    "og:title": "Introduction to streaming for data scientists",
    "og:locale": "en_US",
    "og:description": "As machine learning moves towards real-time, streaming technology is becoming increasingly important for data scientists. Like many people coming from a machine learning background, I used to dread streaming. In our recent survey, almost half of the data scientists we asked said they would like to move from batch prediction to online prediction but can’t because streaming is hard, both technically and operationally. Phrases that the streaming community take for granted like “time-variant results”, “time travel”, “materialized view” certainly don’t help.",
    "og:url": "https://huyenchip.com/2022/08/03/stream-processing-for-data-scientists.html",
    "og:site_name": "Chip Huyen",
    "og:image": "https://huyenchip.com/assets/pics/streaming/transport-compute.png",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "article:published_time": "2022-08-03T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/80b0bf3d0f72/code/body-000.txt",
      "chars": 244,
      "language": "text"
    },
    {
      "file": "attachments/80b0bf3d0f72/code/body-001.txt",
      "chars": 80,
      "language": "text"
    },
    {
      "file": "attachments/80b0bf3d0f72/code/body-002.txt",
      "chars": 204,
      "language": "text"
    },
    {
      "file": "attachments/80b0bf3d0f72/code/body-003.txt",
      "chars": 234,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/1-stateless-processing.png",
      "bytes": 687516,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-fa30d14ec867d5d6024b.png",
      "src": "https://huyenchip.com/assets/pics/streaming/1-stateless-processing.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/real-time-monitoring.png",
      "bytes": 773863,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-693f3b48a61c8dd36589.png",
      "src": "https://huyenchip.com/assets/pics/streaming/real-time-monitoring.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/confluent-origin.png",
      "bytes": 132621,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-e9cbc9cf305c6d70442b.png",
      "src": "https://huyenchip.com/assets/pics/streaming/confluent-origin.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/table-before.png",
      "bytes": 42139,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-54cdb2b6910e6a570bfe.png",
      "src": "https://huyenchip.com/assets/pics/streaming/table-before.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/table-after.png",
      "bytes": 134347,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-e5d2ec61f8901362707f.png",
      "src": "https://huyenchip.com/assets/pics/streaming/table-after.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/transport-compute.png",
      "bytes": 157459,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-a9e26cbfbf8aaf46e675.png",
      "src": "https://huyenchip.com/assets/pics/streaming/transport-compute.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/stateful-processing.png",
      "bytes": 468770,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-98e982e0d88250dd4cc0.png",
      "src": "https://huyenchip.com/assets/pics/streaming/stateful-processing.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/train-predict-inconsistency.png",
      "bytes": 108217,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-5d1b7b05e0c9225cff3f.png",
      "src": "https://huyenchip.com/assets/pics/streaming/train-predict-inconsistency.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/train-predict-consistency.png",
      "bytes": 414845,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-6bfb6ce30d8e0a352c15.png",
      "src": "https://huyenchip.com/assets/pics/streaming/train-predict-consistency.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/featureform.png",
      "bytes": 399902,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-07486e9c3caeb1948f0a.png",
      "src": "https://huyenchip.com/assets/pics/streaming/featureform.png",
      "alt": "Positional features",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/streaming/tecton.png",
      "bytes": 383690,
      "reused": false,
      "local_path": "attachments/80b0bf3d0f72/images/body-9b699c01d4977eece622.png",
      "src": "https://huyenchip.com/assets/pics/streaming/tecton.png",
      "alt": "Positional features",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://huyenchip.com/2022/01/02/real-time-machine-learning-challenges-and-solutions.html",
      "label": "machine learning moves towards real-time",
      "type": "a"
    },
    {
      "href": "https://zhenzhongxu.com/the-four-innovation-phases-of-netflixs-trillions-scale-real-time-data-infrastructure-2370938d7f01",
      "label": "a co-founder who’s super deep into streaming",
      "type": "a"
    },
    {
      "href": "https://blog.codinghorror.com/boyds-law-of-iteration/",
      "label": "Boyd’s Law of Iteration: speed of iteration beats quality of iteration",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/DevOps",
      "label": "speeding up iterations for higher quality",
      "type": "a"
    },
    {
      "href": "https://learning.oreilly.com/library/view/designing-machine-learning/9781098107956/ch09.html",
      "label": "Continual learning",
      "type": "a"
    },
    {
      "href": "https://huyenchip.com/2022/02/07/data-distribution-shifts-and-monitoring.html",
      "label": "data distribution shifts",
      "type": "a"
    },
    {
      "href": "https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying",
      "label": "The log",
      "type": "a"
    },
    {
      "href": "https://twitter.com/jaykreps/status/1408159236794765314",
      "label": "tweet",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Logical_clock",
      "label": "logical clock",
      "type": "a"
    },
    {
      "href": "https://rockset.com/blog/change-data-capture-what-it-is-and-how-to-use-it/",
      "label": "Change Data Capture",
      "type": "a"
    },
    {
      "href": "https://github.com/debezium/debezium",
      "label": "Debezium",
      "type": "a"
    },
    {
      "href": "https://www.fivetran.com/cdc-database-replication",
      "label": "Fivetran",
      "type": "a"
    },
    {
      "href": "https://www.striim.com/blog/change-data-capture-cdc-what-it-is-and-how-it-works/",
      "label": "Striim",
      "type": "a"
    },
    {
      "href": "https://materialize.com/why-use-a-materialized-view/",
      "label": "materialized and incremental materialized",
      "type": "a"
    },
    {
      "href": "https://stackoverflow.com/questions/39882204/airflow-backfill-clarification",
      "label": "Airflow",
      "type": "a"
    },
    {
      "href": "https://www.docs.developers.amplitude.com/analytics/data-backfill-guide/",
      "label": "Amplitude",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.12133",
      "label": "streaming SQL engines",
      "type": "a"
    },
    {
      "href": "https://www.infoq.com/presentations/sql-streaming-apache-flink/",
      "label": "Streaming SQL to Unify Batch & Stream Processing w/ Apache Flink @Uber",
      "type": "a"
    },
    {
      "href": "https://twitter.com/slaterstich/status/1531824582545461248?ref_src=twsrc%5Etfw",
      "label": "June 1, 2022",
      "type": "a"
    },
    {
      "href": "https://docs.google.com/presentation/d/1YtTEnOax5MDA8DazDa1ad-sP4zzM58KQK4HNAcxoONA/edit",
      "label": "open challenges in stream processing",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Chandy%E2%80%93Lamport_algorithm",
      "label": "Chandy–Lamport algorithm",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/CAP_theorem",
      "label": "availability vs. consistency",
      "type": "a"
    },
    {
      "href": "https://twitter.com/luke_metz",
      "label": "Luke Metz",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/chloe-he/",
      "label": "Chloe He",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/hanchunglee/",
      "label": "Han-chung Lee",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/dan-turkel/",
      "label": "Dan Turkel",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/metzgerrobert/",
      "label": "Robert Metzger",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:51Z",
  "source_url": "https://huyenchip.com/2022/08/03/stream-processing-for-data-scientists.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.12133",
      "bytes": 714951,
      "reused": true,
      "local_path": "papers/repair/38bf0a310d229d18fe7b.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 28783,
  "expected_body_images": 11,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/80b0bf3d0f72/pages/source.html",
  "local_html": "attachments/80b0bf3d0f72/pages/article.html",
  "body_base_path": "attachments/80b0bf3d0f72/pages/",
  "body_markdown_path": "attachments/80b0bf3d0f72/pages/article.md"
}