{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Eugene Yan",
  "title_listed": "Real-time Machine Learning For Recommendations",
  "keywords": "实时ML,推荐,流式",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://eugeneyan.com/writing/real-time-recommendations/",
  "publish_date": "2021-01",
  "corpus_stage": "推理部署",
  "notes": "实时机器学习推荐系统",
  "id": "74bd0accd07e",
  "collected_at": "2026-10-02T14:16:57",
  "title": "Real-time Machine Learning For Recommendations",
  "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nReal-time Machine Learning For Recommendations\n[\nteardown\nmachinelearning\nrecsys\nproduction\n🔥\n]\n· 21 min read\nA few weeks ago, Chip compared the state of real-time machine learning in China and US: While many Chinese companies have adopted real-time ML, US companies are still assessing its value. She also wrote about ML going real-time\nhere\n.\nTalking to Internet companies in China gives me the impression that their MLOps infra is away ahead.\nMany US Internet co. are still wondering if there's value in real-time ML. Chinese counterparts are already doing real-time inference + online training.\nAre my samples biased?\n— Chip Huyen (@chipro)\nDecember 10, 2020\nThis post continues the thread and shares\nhow real-time ML looks in practice\n. Drawing from my experience and industry papers/blogs, we’ll discuss\nreal-time recommendations\n.\nWhen does real-time recommendation make sense? When does it not?\nHow have China and US companies implemented real-time recommenders?\nHow can we design and implement a simple MVP?\nNote: This discussion assumes basic knowledge of recommendation systems, such as the difference between item-to-item and user-to-item, and the candidate generation & ranking paradigm. Fret not if those terms are unfamiliar. We’ll have two primers to help you get up to speed. Click on the 👉 below to start with the first.\nPrimer I: Collaboration vs. Content-based, item-to-item vs. user-to-item\nIf you understand the difference between collaboration vs. content-based recommendations, and item-to-item vs. user-to-item, feel free to skip this section.\nCollaboration-based recommendations are based on user behavior\n. Assume I like movie\nX\nand dislike movie\nY\n. To recommend movies to me, we first find users similar to me (i.e., like\nX\n, dislike\nY\n). Then, from those users, what are movies they liked, but I’ve not watched? Those movies are then recommended to me. With user behavioral data, users can “collaborate” to create recommendations for each other. Collaborative filtering is probably the most well-known approach.\nContent-based recommendations are based on item metadata\n. Given the movies I’ve watched (and enjoyed), content-based recommenders suggest movies of similar genre, time period, director, etc. Relative to collaboration-based recommenders, content-based recommenders tend to be more effective when the movie is new and we don’t have enough user behavioral data about it yet (i.e., cold-start problem)\nIn item-to-item (i2i) recommendations, given an item, we recommend other items\n. Here’s an example of\ni2i recommendations on IMDB\n, under the “More Like This” widget. This works well in scenarios where the focus is the item (e.g., item detail page).\nMore like Independence Day on IMDB\nIn user-to-item (u2i), given a user, we recommend items\n. We see this on the home page of Netflix, Amazon, Taobao, sometimes with the name of “Recommended For You”. Our social media feeds (e.g., Twitter, LinkedIn, Facebook) are u2i recommendations too. In such scenarios, the user (and their historical preferences) is the focus.\nRecommended For You on Amazon\nTaken together, i2i and u2i recommendations provide coverage for the bulk of user traffic via detail pages and home pages.\nWhen\n(not)\nto use real-time recommendations?\nBefore we get too excited, let me first say that\nmost use cases won’t need real-time recommendations\n; batch recommendations are good enough.\nRelative to real-time recommendations, batch recommendations are computationally cheaper. They are usually generated once a day and benefit from batch processing’s economies of scale. The recommendations are then loaded into a key-value store (e.g., Redis, DynamoDB) and served via a key-value lookup.\nBatch recommendations are also simpler ops-wise. By caching pre-computed recommendations (in our key-value store), we decouple computation from serving. Thus, even if the compute step fails, there’s no customer-facing impact; we continue to serve the previous batch of (slightly stale) recommendations. The cache provides a buffer and ensures almost 100% uptime, reducing ops burden on the team (e.g., on-call, complaints).\nBatch vs. streaming in the case of analytics (\nsource\n)\nIn contrast, real-time recommendations usually require more computation. For example, we might aggregate streamed events (e.g., click, like, purchase) and generate new recommendations on-demand, based on user interactions. (In comparison, batch recommendations only compute a single set daily.) Furthermore, such computation is done individually and does not benefit from economies of scale. (Nonetheless, we save on not generating recommendations for customers who don’t visit our app).\nOperating real-time recommendations in production is also far tricker. Instead of using ephemeral Spark clusters (for compute) and a DynamoDB (for serving), we’ll need low-latency high-throughput APIs with 24/7 uptime. In the strictest scenario, we won’t have our key-value store as a buffer. The line between compute and serving disappears. Ops burden will increase (read: we’ll be paged in the middle of the night for an outage in another timezone 🚨).\nWhy real-time recommendations then?\nThey’re useful when the customer journey is mission-centric and depends on the context. Such missions are often time-sensitive. Real-time demand fades quickly; demand could be met (on a competitor site) or the user might lose interest. We’ll examine this in two examples: shopping and watching a movie.\nShopping is a\nmission-centric activity\n. Though a customer might predominately purchase a category of products, their shopping behavior is often punctuated with tangential missions. For example, if I usually shop for clothes, my u2i recommendations will mostly be fashion recommendations. However, if I need a new wide-screen monitor and start browsing for one, my u2i recommendations should update ASAP to help me quickly fulfill my mission (lest I go to a competing app).\nIn this scenario, batch recommendations don’t react fast enough. And even when the recommendations are updated, due to the data imbalance, such mission-related needs are not met (More in Section 2.1 of this\nNetflix paper\n).\nThe movies we watch\ndepend on context\n(though our long-term preferences are fairly stable). For example, we might watch different movies depending on whether we’re alone, with friends, with a romantic interest, or with children. It might also depend on our mood, as well as the time of day. Similar to shopping, daily batch recommendations face the same challenges here.\nOther than the examples above, real-time recommendations are also handy in:\nTravel (our vacation destinations are often changing)\nYouTube (we watch videos for coding, yoga, and recipes within the same day)\nServing ads (our interest is time-sensitive; our attention spans are shrinking)\nReal-time recommendations are also useful when the majority of our customers are new (i.e., cold-start). This happens when we’re in the customer acquisition stage, such as when we’ve just launched a new product or entered a new market (e.g., e-commerce in Southeast Asia in 2013 - 2015).\nImagine you’ve just downloaded an e-commerce app. Since we’re uncertain of your gender, the home page will have a mix of categories catering to each gender, from dresses to men’s shirts, from GPUs to makeup. If you click on a dress, we can immediately build a persona (that you’re female) and personalize your shopping experience. In this case, the u2i recommendations on your home page will tilt towards female products.\nGiven that\n1 in 4 users abandon mobile apps\nafter only one use\n, quickly responding to customer needs—from the very first touchpoint—helps with acquisition and retention.\nPrimer II: Candidate generation and ranking\nDid you miss the first primer (collaborative vs. content-based, item-to-item vs. user-to-item)? Click on the  👉\nhere\n.\nMost modern recommenders have two key components: candidate generation and ranking.\nCandidate generation is a fast—but coarse—approach to get (hundreds of) item candidates from millions of items. We trade off precision for efficiency to reduce the search space (e.g., from 100 million to 1,000 candidates, a 99.999% reduction). This is usually done via metadata-based filters (e.g., category, brand) or k-nearest neighbors.\nRanking is a slower—but more precise—approach to sort and select top recommendation candidates. We have leeway to include features that might not have been feasible in the candidate generation step. Such features include user persona (e.g., demographics, price propensity), item metadata (e.g., attributes, engagement statistics), cross features (e.g., interaction between each feature pair), and media embeddings.\nRanking can be framed as either a classification or learning to rank problem. As a classification problem, we can score candidates based on probability of click or purchase. Logistic regression with crossed features is simple to implement and a difficult baseline to beat. Decision trees are also commonly used. As a learning to rank problem, commonly used algorithms include\nLambdaMart\n,\nXGBoost\n, and\nLightGBM\n. Neural networks are also gaining adoption, thanks to\ngains in model efficiency\nvia distillation, pruning, quantization, etc.\nIndustry examples of real-time recommendations\nThis\ngeneral reference\nsheds light on recommendation systems from a Chinese perspective. It’s organized based on the paradigm discussed, with sections for candidate generation (“Match”) and ranking (“Rank”). It also discusses two other components: profile (building user preferences) and post-processing (e.g., excluding previously purchased goods, item-deduplication across recommendation sets).\nTo see how recommendations can be incrementally updated\n, we’ll discuss two algorithms: the humble collaborative filtering (CF) and Alibaba’s Swing algorithm that improves on CF.\nCollaborative filtering (for i2i recommendations) is implemented via the three formulas below. Let’s try to understand what they do and the intuition behind them.\n\\[\\omega_{user}=\\frac{1}{\\left(\\log _{2}(3+cnt)\\right)^{2}}\\]\n\n\\[\\omega_{\\text {item }}=\\sum_{\\text {user }} \\omega_{\\text {user }}\\]\n\n\\[\\operatorname{score}(i, j)=\\sum_{u \\operatorname{ser}} \\frac{\\omega_{user}}{1+\\left(\\omega_{i} * \\omega_{j}\\right)^{0.5}}\\]\nUser weight\n: A user’s weight is inversely proportionate to their number of item interactions (e.g., click, like, wish-listed). Intuition: Users who browse a lot tend to be less discriminative and thus have more noisy behavioral data.\nItem weight\n: The more users interact with an item, the higher its weight.\nItem similarity score\n: To calculate similarity between a pair of items, we sum the weight of users who interacted with both items, and divide it by the product of item weights. Intuition: The greater the overlap in user-interactions, the higher the item similarity (similar to collaborative filtering via matrix factorization or\nalternating least squares\n).\nThe Swing algorithm differs in that the weight (of a pair of users) depends on the number of items they both interacted with (in contrast, CF keeps user weights constant). The greater the proportion of items both users interacted with (i.e., intersection), the higher the user weights.\n\\[\\omega_{u 1}=\\frac{1}{(cnt+5)^{0.35}}\\]\n\n\\[\\omega_{\\text {pair }}=\\omega_{u 1} * \\omega_{u 2}\\]\n\n\\[\\operatorname{score}(i, j)=\\sum_{\\text {pair }} \\frac{\\omega_{\\text {pair }}}{1+\\text { intersection }}\\]\nUser weight\n: Similar to collaborative filtering.\nUser-pair weight\n: Product of user weights. For\nn\nusers, we have\nn^2\nuser-pair weights.\nItem similarity score\n: Similar to collaborative filtering. The key difference is that the denominator\nonly\nconsiders the intersection of products\nboth\nusers have interacted with. (In contrast, CF considers all products any user has interacted with).\nTo make things clearer, here’s how we would calculate item similarity in code:\nfor\ni\nin\nxrange\n(\n0\n,\nlen\n(\nu2items\n)):\nwi\n=\nmath\n.\npow\n(\nlen\n(\nu2items\n[\ni\n])\n+\n5\n,\n-\n0.35\n)\nfor\nj\nin\nxrange\n(\ni\n+\n1\n,\nlen\n(\nu2items\n)):\nintersection\n=\nu2items\n[\ni\n]\n&\nu2items\n[\nj\n]\nwj\n=\nwi\n*\nmath\n.\npow\n(\nlen\n(\nu2items\n[\nj\n])\n+\n5\n,\n-\n0.35\n)\nfor\nproduct_id\nin\nintersection\n:\ni2i\n[\nproduct_id\n]\n=\ni2i\n.\nget\n(\nproduct_id\n,\n0.0\n)\n+\nwj\n/\n(\n1\n+\nlen\n(\nintersection\n))\n# u2items = array of users and their items\n# u2items[i] = items user i clicked on\n# u2items[j] = items user j clicked on\n# intersection = items both user i and user j clicked on\n# wj = product-pair score\n# i2i is incrementally updated as we loop through users (we won't use a loop in production)\nIntuitively, if two users with very different tastes (and product interactions) click on the same product-pair, this suggests a strong relationship between the product-pair. Conversely, if two users have similar tastes (and many product pairs), the product-pair relationship is weaker. This reduces the noise from herd behavior (e.g.,\nHarry Potter problem\n) and identifies more meaningful i2i recommendations.\nFrom these simple equations, we see how recommendations can be\nincrementally updated in real-time\n. Instead of batch matrix multiplication, user and item weights are updated with each customer interaction, perhaps in a key-value store. The weights can then be combined—via summation—to update i2i recommendations.\nThese equations can also be adapted to calculate user affinity towards category, brand, seller, price point, etc. Such affinities are then be used to weigh recommendation candidates. For example, if a user has historical preference for a specific brand, we give items with that brand a higher weight.\nHere’s an example of Swing and category affinity used in\nreal-time recommendations at Alibaba\n. This implementation is for the “Recommended For You” widget on the home page of\n1688\n, a B2B e-commerce. The widget receives clicks from 72% of users.\nReal-time recommendations architecture for Alibaba 1688 (\nsource\n)\nA quick rundown of the infra components:\niGraph\n: Distributed graph database. Think of it as a key-value store, where nodes and edges are keys, and weights are values. Allows for efficient node-edge joins and graph queries. Used to store user preference and item metadata (e.g., user and item weights from the algorithms discussed).\nABFS\n: Alibaba basic feature server. Used for real-time computation of statistical features on user interactions (e.g., clicks, likes, purchases, etc.)\nBE\n: Basic engine. Highly efficient item-level filters and embedding-based kNN.\nUsed for candidate generation.\nRTP\n: Real-time prediction. Models for scoring item via various objectives (e.g., CTR, conversion). Includes logistic regression,\nWide & Deep\n,\nProduct-based Neural Network\n, and\nBehavioral Sequence Transformer\n.\nUsed for ranking.\nPorsche\n: Distributed model training. Used in offline training and validation. Can also be used to update model parameters online, though not applied here.\nTPP\n:\nTaobao\npersonalization platform. Abstraction layer for the infra components above to improve developer productivity and experience.\nHere’s how recommendations are updated and served. As the user browses on the app:\nWith each item interaction (e.g., click, like, add-to-cart, purchase), ABFS computes user and item statistical features. These are passed to BE, and optionally updated in iGraph asynchronously.\nBE generates the top 1,000 candidates based on user and item features (i.e., user preference, item trends) from ABFS. In the image, we see BE using the Swing i2i and c2i (category-to-item) algorithms.\nRTP ranks the 1,000 candidates. Features can be added at this stage, such as user profile (gender, age, price propensity), item attributes (category, brand, seller), context (match with user last click/search), cross features (i.e., interaction features), and sequence features (click/category sequence). The top 600 products are presented to the user.\nTencent’s approach\nis similar.\nThe crux is to break up item-based CF into two aggregates: item count and pair count. Item-to-item similarity is then computed using these aggregates. Thus, item and pair aggregates can be incrementally updated and combined to generate recommendations in real-time. Their system is implemented on\nApache Storm\nand used in Tencent News, Tencent Videos, YiXun (e-commerce), and QQ ads (messaging).\n\\[\\text { itemCount }\\left(i_{p}\\right)=\\sum r_{u, p}\\]\n\n\\[\\operatorname{pairCount}\\left(i_{p}, i_{q}\\right)=\\sum_{u \\in U} \\operatorname{co-rating}\\left(i_{p}, i_{q}\\right)\\]\n\n\\[\\operatorname{sim}\\left(i_{p}, i_{q}\\right)=\\frac{\\operatorname{pairCount}\\left(i_{p}, i_{q}\\right)}{\\sqrt{\\operatorname{itemCount}\\left(i_{p}\\right)} \\sqrt{\\operatorname{itemCount}\\left(i_{q}\\right)}}\\]\nA/B tests showed that real-time recommendations led to 6 - 8% increase in CTR for news recommendations, and 6 - 18% increase in CTR for e-commerce recommendations.\nThe paper also shares tricks used to solve several challenges (listed below). Highly recommended read.\nHoeffding bounds to prune redundant user-interaction events\nDemographic-based clustering to tackle cold-start problems\nCaching to address high reads due to traffic spikes\nBuffers to address data skews (referred to as the “hot item problem”)\nNext, we look west and see how US companies have implemented real-time recommendations.\nFirst, here’s\nYouTube’s video recommendation system\n. We should be familiar with the overall design paradigm by now.\nReal-time recommendation architecture for YouTube (\nsource\n)\nFor candidate generation, approximate kNN is applied on video and search embeddings to select hundreds of videos from millions (probably billions by now).\nFor ranking, a deep neural network is used to score each video and select the best couple dozen. Additional features (e.g., query and video statistics, previous user interaction) help improve precision. The neural network is trained via weighted logistic regression, where positive labels are weighted by video watch time (negative labels are unweighted). This reflects the business objective of increasing expected watch time per video impression.\nInstagram’s\napproach\nis similar\n. In the candidate generation stage, account embeddings are used to identify accounts similar to those the user has previously interacted with. From these account candidates, they sample 500 media candidates (e.g., photos, stories).\nReal-time recommendation architecture for Instagram (\nsource\n)\nThen, these media candidates go through a three-pass ranking process which uses a combination of techniques to shrink neural network models:\nFirst pass: A distilled model mimics the later stages with minimal features to return the top 150 ranked candidates.\nSecond pass: A lightweight neural network uses the full set of dense features and returns the top 50 ranked candidates.\nFinal pass: A deep neural network uses the full set  of dense and sparse features to return the top 25 ranked candidates (for the first page of the Explore grid).\nFor our final example, we look at\nhow Netflix re-ranks recommendations in real-time\n.\nOn most Netflix pages, we’ll see several recommendation rows. Each recommendation row has dozens of movies. The goal is to reorder the recommendation rows (i.e., North-South ordering) as well as the movies within each recommendation row (i.e., East-West order).\nRecommendation rows on Netflix's home screen (\nsource\n)\nHorizontal scrolls (on the recommendation rows) are used as user-interaction signals. They showed that video play probability is positively correlated with the number of horizontal scrolls.\nRelationship between horizontal scrolling and watch probability (\nsource\n)\nIn contrast to previous examples, re-ranking is carried out\non the client device\n, avoiding a roundtrip to the server. (However, this also prevents candidate generation.) Given that viewing devices can be low-powered (e.g., TVs), computation needs to be lightweight. Thus, the algorithms used are simple—yet effective—probabilistic models. This allows recommendation rows below the fold (i.e., not on screen) to be updated on-the-fly as users perform horizontal scrolling above the fold.\nThus, we see that real-time recommendations are increasingly common, both in China and US. However, are they\nonly\navailable to big tech?\nNo.\nHere’s how to cheaply build an MVP.\nHow to design and implement an MVP\nAfter the examples above, you’re might think real-time recommenders require specialized infra, deep learning models, and all that jazz.\nI hope to convince you otherwise.\nWe’ll briefly go through how to design and build an MVP, focusing on what’s commonly viewed as the bottleneck of real-time recommenders:\ncompute and serving\n. In contrast, training is relatively easier and widely discussed.\nTo begin, I think it’s useful to approach DS/ML systems in three broad strokes:\nRequirements (or constraints)\n: What does success look like? What can we not do?\nMethodology\n: How will we use data and code to achieve success?\nImplementation\n: What infrastructure is needed in production?\nWhen defining requirements, we should start from the customer. How will real-time recommendations improve the customer experience, and in turn benefit the business? What business metrics are important? Goals and success metrics will vary based on the business and use case, and will not be defined here.\nFor our MVP, perhaps more important than requirements are\nconstraints\n(i.e., how\nnot\nto solve the problem). Here are some constraints for our real-time recommender:\nLatency: App responsiveness is key to user experience. Google found that taking an additional 500ms to generate search results\nreduced traffic by 20%\n. Amazon shared that 100ms additional latency\nreduces profit by 1%\n. Thus, for our MVP, we set a budget of\n200ms latency\n(excluding network latency from users to our server).\nThroughput: To assess our design’s scalability, we set an expected throughput of\n1,000 queries per second (QPS)\n, or about 84 million queries a day. We probably won’t need 1,000 QPS throughout the day as traffic fluctuates.\nCost: To achieve a healthy ROI, cost should be a fraction of expected revenue. Assuming real-time recommendations reap an additional 100k monthly, we set an\ninfra budget of 10k monthly\n(i.e., 10%).\nWe should also consider other aspects such as availability (aka redundancy), security, privacy, ethics, etc. Nonetheless, for our MVP, the first three constraints are technical and business showstoppers which we’ll focus on.\nTo train item embeddings, we adopt the simple but effective word2vec approach\n, specifically, the skip-gram model. (This is also used by\nInstagram\n,\nTwitter\n, and\nAlibaba\n.) I’ve\npreviously written\nabout how to create embeddings via word2vec and DeepWalk and won’t go into details here.\nTo generate candidates, we apply k-nearest neigbours\n(à la YouTube’s implementation). However, exact kNN is slow and we don’t really need the precision at this stage. Thus, we’ll use\napproximate\nnearest neighbours\n(ANN) instead.\nThere are several open-sourced ANN implementations, such as\nFacebook’s FAISS\n,\nGoogle’s ScANN\n, and Hierarchical Navigable Small Word Graphs (\nhnswlib\n). We’ll benchmark them on the recall/latency trade-off. To mimic production conditions, each query consists of a single embedding (i.e., batch size = 1). The graph below shows ScaNN outperforming the other two implementations. (FAISS, in particular, is\noptimized for batch queries\n.)\nBenchmarking ANNs on recall vs latency; top-right is better.\nInstalling ScaNN on Mac is a pain!\nInstalling ScaNN on a Mac is tricky and took a few frustrating hours to figure out. If you run into problems, you might find this\nstep-by-step\nhelpful.\nIn addition, here are some GitHub issues that might help:\nTF_SHARED_LIBRARY_NAME set with typos when configure\nError: Reason: image not found\nBuilding from source shows error (undefined symbol: _Py_ZeroStruct)\nTo rank candidates, we start with a single-layer neural network\n(read: logistic regression) with cross features—simple, yet difficult to beat. Cross features are created by combining all possible pairs of features to capture their interactions. While this blows up the feature space, it’s not a problem for logistic regression.\nBeyond machine learning metrics (i.e., recall@k, NDCG, AUC), it’s probably more important to consider business metrics (circling back to requirements). Different goals call for different metrics:\nIf our goal is\nincreased engagement\n, or to sell ads based on clicks, we’ll want to consider absolute clicks, click-thru-rate, daily average users.\nIf our goal is\ncustomer acquisition\n, we’ll want to optimize for first sale (e.g., units sold, conversion), monthly average users, and be willing to take a hit on revenue.\nIf our goal is\nincreased revenue\n, we’ll want to focus on revenue per session, average basket size, and customer lifetime value.\nFrom experience, business stakeholders will usually have conflicting goals. Marketing wants to make the first sale (regardless of item price), customer experience wants to takedown poor quality products (even if they sell well), and commercial wants to maximize profit (by selling higher-priced items). Getting everyone to agree on key metrics and guardrails can be more difficult than improving our models.\nWill our MVP require specialized infrastructure? Not necessarily.\nA cost-effective approach is to use\nEC2 instances\nthat can scale horizontally with a\nload balancer\nin front. To further simplify things, we can just use\nAWS SageMaker\n.\nSageMaker takes care of load-balancing on multiple instances (\nimage source\n)\nTo assess latency and throughput, we have various options including\nserverless-artillery\n(\nAWS guide\n) and\nlocust\n. Running several load tests showed that a SageMaker endpoint backed by 30 m5.xlarge instances was able to serve 1,200 queries per second without breaking a sweat. At this throughput, median latency was 25ms while the 99th percentile was 65ms. There were zero errors.\nWith regard to cost, m5.xlarge (16 gb RAM, 4 CPUs) instances in US West (Oregon) have an\nhourly rate of $0.269\n. Running 30 instances for 28 days works out to approximately 5.5k, well within our 10k budget. Using reserved instances and/or auto-scaling can help with lowering cost.\nOur simple MVP deliberately excludes several considerations. For example, to let other services query our item embeddings, we might want to expose them as a separate service. This will require additional infra (e.g., DynamoDB) which will increase cost and ops burden. The additional service call (for item embeddings) will also add latency (10-30ms) though it can be minimized via a good network setup. Also, how can we expose our candidate generation and ranking services via generic APIs, so other users can mix-and-match as required? We’ll want to consider these in the long-term roadmap.\nBuilding a real-time recommender need not be hard\nI hope this improves your understanding of real-time machine learning in the context of recommendation systems, and demonstrates that it’s\nnot\nan insurmountable challenge. Libraries (e.g., ScaNN) and managed services (e.g., AWS SageMaker) abstract away much of the nitty-gritty such as optimization, health checks, auto-scaling, recovery, etc. Building on them allows for effective, low-cost, real-time ML.\nDesigning and implementing machine learning systems can be difficult. If you’re building one and would like another pair of eyes or feedback, reach out via\n@eugeneyan\nor\nemail\n.\nHow does real-time ML look in practice?\nWe discuss real-time recommenders, drawing on my experience & industry papers:\n• When does it make sense? When does it not?\n• How have China & US companies implemented them?\n• How do we design & build an MVP?\nhttps://t.co/hSfRr3q2ie\n— Eugene Yan (@eugeneyan)\nJanuary 12, 2021\nReferences\n推荐系统技术\n机器如何“猜你喜欢”？深度学习模型在1688的应用实践\nTencentRec: Real-time Stream Recommendation in Practice\nDeep Neural Networks for YouTube Recommendations\nPowered by AI: Instagram’s Explore recommender system\nUsing Navigation to Improve Recommendations in Real-Time\n为电商而生的知识图谱，如何感应用户需求\nThe Secret Behind Taobao’s AI-Powered Personalized Recommendations\nPersonalized Bundle List Recommendation\nEmbeddings@Twitter\nBillion-scale Commodity Embedding for Recommendation in Alibaba\nMore recommendation systems papers in the\napplied-ml\nrepository\nThanks\nto Yang Xinyi,\nKarl Higley\n,\nWill Larson\n, and\nTushar Chandra\nfor reading drafts of this. Thanks to\nHammad Khan\nfor a correction.\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Jan 2021). Real-time Machine Learning For Recommendations. eugeneyan.com.\n        https://eugeneyan.com/writing/real-time-recommendations/.\nor\n@article{yan2021realttime,\n  title   = {Real-time Machine Learning For Recommendations},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2021},\n  month   = {Jan},\n  url     = {https://eugeneyan.com/writing/real-time-recommendations/}\n}\nShare on:\nBrowse related tags:\n[\nteardown\nmachinelearning\nrecsys\nproduction\n🔥\n]\nor\nSearch\n« 2021 Roadmap: Sharing, Helping, and Living More\nMailbag: How to Get Experienced DS Resume Noticed by Recruiters? »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates",
  "metadata": {
    "title": "Real-time Machine Learning For Recommendations",
    "description": "Why real-time? How have China & US companies built them? How to design & build an MVP?",
    "keywords": null,
    "author": "Eugene Yan",
    "og:site_name": "eugeneyan.com",
    "og:title": "Real-time Machine Learning For Recommendations",
    "og:type": "article",
    "og:description": "Why real-time? How have China & US companies built them? How to design & build an MVP?",
    "og:url": "https://eugeneyan.com/writing/real-time-recommendations/",
    "og:image": "https://eugeneyan.com/assets/og_image/real-time.jpg",
    "twitter:card": "summary_large_image",
    "twitter:domain": "eugeneyan.com",
    "twitter:title": "Real-time Machine Learning For Recommendations",
    "twitter:description": "Why real-time? How have China & US companies built them? How to design & build an MVP?",
    "twitter:image": "https://eugeneyan.com/assets/og_image/real-time.jpg",
    "article:published_time": "2021-01-10T00:00:00+00:00",
    "article:author": "https://eugeneyan.com/about/",
    "article:section": "posts",
    "article:tag": "🔥"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/74bd0accd07e/code/body-000.txt",
      "chars": 648,
      "language": "text"
    },
    {
      "file": "attachments/74bd0accd07e/code/body-001.txt",
      "chars": 255,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-search.svg",
      "bytes": 425,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-feeb4bd87b8704a903b2.svg",
      "src": "https://eugeneyan.com/assets/icon-search.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/mlt.webp",
      "bytes": 33548,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-30e44d64cc5eede6b62d.webp",
      "src": "https://eugeneyan.com/assets/mlt.webp",
      "alt": "More like Independence Day on IMDB",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/rfy.webp",
      "bytes": 21880,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-b66d855c07e663371955.webp",
      "src": "https://eugeneyan.com/assets/rfy.webp",
      "alt": "Recommended For You on Amazon",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/batch-vs-stream.webp",
      "bytes": 25406,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-3845f05bdd70666dfa52.webp",
      "src": "https://eugeneyan.com/assets/batch-vs-stream.webp",
      "alt": "Batch vs. streaming in the case of analytics",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/1688-architecture.webp",
      "bytes": 15508,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-3e74dfcd129eddf59fab.webp",
      "src": "https://eugeneyan.com/assets/1688-architecture.webp",
      "alt": "Real-time recommendations on Alibaba 1688",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/youtube-architecture.webp",
      "bytes": 16898,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-351d383124d16fcb082e.webp",
      "src": "https://eugeneyan.com/assets/youtube-architecture.webp",
      "alt": "Real-time recommendations on YouTube",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/instagram-architecture.gif",
      "bytes": 1402404,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-f4571b46c51c9a58acd2.gif",
      "src": "https://eugeneyan.com/assets/instagram-architecture.gif",
      "alt": "Real-time recommendations on Instagram",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/netflix-home.webp",
      "bytes": 32376,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-ce7ef56f5bf266f78c10.webp",
      "src": "https://eugeneyan.com/assets/netflix-home.webp",
      "alt": "Recommendation rows on Netflix's home screen",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/netflix-horizontal-scroll.webp",
      "bytes": 47832,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-f6e60ccd366490ed061c.webp",
      "src": "https://eugeneyan.com/assets/netflix-horizontal-scroll.webp",
      "alt": "Relationship between horizontal scrolling and watch probability",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/ann-benchmarks.webp",
      "bytes": 12254,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-0ec55c4fba58b26c46f5.webp",
      "src": "https://eugeneyan.com/assets/ann-benchmarks.webp",
      "alt": "Benchmarking ANNs on recall vs latency",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/load-balancer.gif",
      "bytes": 1139741,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-f6d3d8414486dcfa2ae9.gif",
      "src": "https://eugeneyan.com/assets/load-balancer.gif",
      "alt": "Load balancing with multiple instances",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-twitter.svg",
      "bytes": 892,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-0083a7581b07e5f40e7b.svg",
      "src": "https://eugeneyan.com/assets/icon-twitter.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "bytes": 559,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-f7a16c3e46e5838cec4d.svg",
      "src": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bluesky.svg",
      "bytes": 745,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-be754e8ecdaf6de94f55.svg",
      "src": "https://eugeneyan.com/assets/bluesky.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-facebook.svg",
      "bytes": 1123,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-5d55b59733923cf14223.svg",
      "src": "https://eugeneyan.com/assets/icon-facebook.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-mail.svg",
      "bytes": 841,
      "reused": false,
      "local_path": "attachments/74bd0accd07e/images/body-b179147617228dfafb9f.svg",
      "src": "https://eugeneyan.com/assets/icon-mail.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://eugeneyan.com/",
      "label": "eugeneyan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/start-here/",
      "label": "Start Here",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/",
      "label": "Writing",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/",
      "label": "Speaking",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/prototyping/",
      "label": "Prototyping",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/search/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/teardown/",
      "label": "teardown",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/machinelearning/",
      "label": "machinelearning",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/recsys/",
      "label": "recsys",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/production/",
      "label": "production",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/🔥/",
      "label": "🔥",
      "type": "a"
    },
    {
      "href": "https://huyenchip.com/2020/12/27/real-time-machine-learning.html",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://twitter.com/chipro/status/1337077324936663040?ref_src=twsrc%5Etfw",
      "label": "December 10, 2020",
      "type": "a"
    },
    {
      "href": "https://www.imdb.com/title/tt0116629/",
      "label": "i2i recommendations on IMDB",
      "type": "a"
    },
    {
      "href": "https://softwareengineeringdaily.com/2018/11/09/converged-data-platform-unifying-streaming-data-using-mapr/",
      "label": "source",
      "type": "a"
    },
    {
      "href": "https://dl.acm.org/doi/10.1145/3240323.3240372",
      "label": "Netflix paper",
      "type": "a"
    },
    {
      "href": "https://techcrunch.com/2016/05/31/nearly-1-in-4-people-abandon-mobile-apps-after-only-one-use/",
      "label": "1 in 4 users abandon mobile apps",
      "type": "a"
    },
    {
      "href": "https://www.microsoft.com/en-us/research/publication/ranking-boosting-and-model-adaptation/",
      "label": "LambdaMart",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1603.02754",
      "label": "XGBoost",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html",
      "label": "LightGBM",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/notes-from-sparkai-summit-application-agnostic/",
      "label": "gains in model efficiency",
      "type": "a"
    },
    {
      "href": "https://blog.csdn.net/xwd18280820053/article/details/90316416",
      "label": "general reference",
      "type": "a"
    },
    {
      "href": "https://spark.apache.org/docs/latest/ml-collaborative-filtering.html",
      "label": "alternating least squares",
      "type": "a"
    },
    {
      "href": "https://www.quora.com/Recommendation-Systems-What-exactly-is-Harry-Potter-Problem",
      "label": "Harry Potter problem",
      "type": "a"
    },
    {
      "href": "https://102.alibaba.com/detail?id=183",
      "label": "real-time recommendations at Alibaba",
      "type": "a"
    },
    {
      "href": "https://www.1688.com/index.html",
      "label": "1688",
      "type": "a"
    },
    {
      "href": "https://ai.googleblog.com/2016/06/wide-deep-learning-better-together-with.html",
      "label": "Wide & Deep",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1611.00144",
      "label": "Product-based Neural Network",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.06874",
      "label": "Behavioral Sequence Transformer",
      "type": "a"
    },
    {
      "href": "https://world.taobao.com/",
      "label": "Taobao",
      "type": "a"
    },
    {
      "href": "https://dl.acm.org/doi/10.1145/2723372.2742785",
      "label": "Tencent’s approach",
      "type": "a"
    },
    {
      "href": "https://storm.apache.org/",
      "label": "Apache Storm",
      "type": "a"
    },
    {
      "href": "https://dl.acm.org/doi/10.1145/2959100.2959190",
      "label": "YouTube’s video recommendation system",
      "type": "a"
    },
    {
      "href": "https://ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommender-system/",
      "label": "approach",
      "type": "a"
    },
    {
      "href": "https://dl.acm.org/doi/10.1145/2959100.2959174",
      "label": "how Netflix re-ranks recommendations in real-time",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/what-i-do-before-a-data-science-project-to-ensure-success/",
      "label": "constraints",
      "type": "a"
    },
    {
      "href": "https://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html",
      "label": "reduced traffic by 20%",
      "type": "a"
    },
    {
      "href": "http://radar.oreilly.com/2008/08/radar-theme-web-ops.html",
      "label": "reduces profit by 1%",
      "type": "a"
    },
    {
      "href": "https://blog.twitter.com/engineering/en_us/topics/insights/2018/embeddingsattwitter.html",
      "label": "Twitter",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1803.02349",
      "label": "Alibaba",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/recommender-systems-graph-and-nlp-pytorch/",
      "label": "previously written",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Nearest_neighbor_search",
      "label": "approximate nearest neighbours",
      "type": "a"
    },
    {
      "href": "https://engineering.fb.com/2017/03/29/data-infrastructure/faiss-a-library-for-efficient-similarity-search/",
      "label": "Facebook’s FAISS",
      "type": "a"
    },
    {
      "href": "https://ai.googleblog.com/2020/07/announcing-scann-efficient-vector.html",
      "label": "Google’s ScANN",
      "type": "a"
    },
    {
      "href": "https://github.com/nmslib/hnswlib",
      "label": "hnswlib",
      "type": "a"
    },
    {
      "href": "https://github.com/facebookresearch/faiss/wiki/FAQ",
      "label": "optimized for batch queries",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/how-to-install-scann-on-mac/",
      "label": "step-by-step",
      "type": "a"
    },
    {
      "href": "https://github.com/google-research/google-research/issues/342",
      "label": "TF_SHARED_LIBRARY_NAME set with typos when configure",
      "type": "a"
    },
    {
      "href": "https://github.com/google-research/google-research/issues/363",
      "label": "Error: Reason: image not found",
      "type": "a"
    },
    {
      "href": "https://github.com/google-research/google-research/issues/402",
      "label": "Building from source shows error (undefined symbol: _Py_ZeroStruct)",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/ec2/instance-types/",
      "label": "EC2 instances",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/elasticloadbalancing/",
      "label": "load balancer",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/sagemaker/",
      "label": "AWS SageMaker",
      "type": "a"
    },
    {
      "href": "https://medium.com/dazn-tech/aws-application-load-balancer-algorithms-765be2eca158",
      "label": "image source",
      "type": "a"
    },
    {
      "href": "https://github.com/Nordstrom/serverless-artillery",
      "label": "serverless-artillery",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/blogs/machine-learning/load-test-and-optimize-an-amazon-sagemaker-endpoint-using-automatic-scaling/",
      "label": "AWS guide",
      "type": "a"
    },
    {
      "href": "https://locust.io/",
      "label": "locust",
      "type": "a"
    },
    {
      "href": "https://aws.amazon.com/sagemaker/pricing/",
      "label": "hourly rate of $0.269",
      "type": "a"
    },
    {
      "href": "https://twitter.com/eugeneyan",
      "label": "@eugeneyan",
      "type": "a"
    },
    {
      "href": "https://t.co/hSfRr3q2ie",
      "label": "https://t.co/hSfRr3q2ie",
      "type": "a"
    },
    {
      "href": "https://twitter.com/eugeneyan/status/1348803884194484225?ref_src=twsrc%5Etfw",
      "label": "January 12, 2021",
      "type": "a"
    },
    {
      "href": "https://102.alibaba.com/detail?id=179",
      "label": "为电商而生的知识图谱，如何感应用户需求",
      "type": "a"
    },
    {
      "href": "https://medium.com/datadriveninvestor/the-secret-behind-taobaos-ai-powered-personalized-recommendations-c74398c64f3d",
      "label": "The Secret Behind Taobao’s AI-Powered Personalized Recommendations",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.01933",
      "label": "Personalized Bundle List Recommendation",
      "type": "a"
    },
    {
      "href": "https://github.com/eugeneyan/applied-ml",
      "label": "applied-ml",
      "type": "a"
    },
    {
      "href": "https://twitter.com/karlhigley",
      "label": "Karl Higley",
      "type": "a"
    },
    {
      "href": "https://lethain.com/",
      "label": "Will Larson",
      "type": "a"
    },
    {
      "href": "https://tusharc.dev/",
      "label": "Tushar Chandra",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/hammad-khan-b84822142/",
      "label": "Hammad Khan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/2021-roadmap/",
      "label": "« 2021 Roadmap: Sharing, Helping, and Living More",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/mailbag-resume-for-experienced-ds/",
      "label": "Mailbag: How to Get Experienced DS Resume Noticed by Recruiters? »",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:19Z",
  "source_url": "https://eugeneyan.com/writing/real-time-recommendations/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1603.02754",
      "bytes": 944793,
      "reused": false,
      "local_path": "papers/repair/e3e654cf601abac0d81a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1611.00144",
      "bytes": 481578,
      "reused": false,
      "local_path": "papers/repair/f72f43aa3a3908e53be5.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.06874",
      "bytes": 1086464,
      "reused": true,
      "local_path": "papers/repair/717557f2835b842586d2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1803.02349",
      "bytes": 2971166,
      "reused": true,
      "local_path": "papers/repair/3f46e0176c41018652f3.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.01933",
      "bytes": 1990737,
      "reused": false,
      "local_path": "papers/repair/7ca7e18ee3511700f1ef.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 29230,
  "expected_body_images": 16,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/74bd0accd07e/pages/source.html",
  "local_html": "attachments/74bd0accd07e/pages/article.html",
  "body_base_path": "attachments/74bd0accd07e/pages/",
  "body_markdown_path": "attachments/74bd0accd07e/pages/article.md"
}