{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Chip Huyen",
  "title_listed": "Building A Generative AI Platform",
  "keywords": "生成式AI,平台,架构",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huyenchip.com/2024/07/25/genai-platform.html",
  "publish_date": "2024-07",
  "corpus_stage": "推理部署",
  "notes": "生成式AI平台架构设计",
  "id": "2baeb4feb113",
  "collected_at": "2026-10-02T14:14:11",
  "title": "Building A Generative AI Platform",
  "body_markdown": "After studying how companies deploy generative AI applications, I noticed many similarities in their platforms. This post outlines the common components of a generative AI platform, what they do, and how they are implemented. I try my best to keep the architecture general, but certain applications might deviate. This is what the overall architecture looks like.\nThis is a pretty complex system. This post will start from the simplest architecture and progressively add more components. In its simplest form, your application receives a query and sends it to the model. The model generates a response, which is returned to the user. There are no guardrails, no augmented context, and no optimization. The\nModel API\nbox refers to both third-party APIs (e.g., OpenAI, Google, Anthropic) and self-hosted APIs.\nFrom this, you can add more components as needs arise. The order discussed in this post is common, though you don’t need to follow the exact same order. A component can be skipped if your system works well without it. Evaluation is necessary at every step of the development process.\nEnhance context input into a model by giving the model access to external data sources and tools for information gathering.\nPut in guardrails to protect your system and your users.\nAdd model router and gateway to support complex pipelines and add more security.\nOptimize for latency and costs with cache.\nAdd complex logic and write actions to maximize your system’s capabilities.\nObservability, which allows you to gain visibility into your system for monitoring and debugging, and orchestration, which involves chaining all the components together, are two essential components of the platform. We will discuss them at the end of this post.\n» What this post is not «\nThis post focuses on the overall architecture for deploying AI applications. It discusses what components are needed and considerations when building these components. It’s not about how to build AI applications and, therefore, does NOT discuss model evaluation, application evaluation, prompt engineering, finetuning, data annotation guidelines, or chunking strategies for RAGs. All these topics are covered in my upcoming book\nAI Engineering\n.\nStep 1. Enhance Context\nThe initial expansion of a platform usually involves adding mechanisms to allow the system to augment each query with the necessary information. Gathering the relevant information is called context construction.\nMany queries require context to answer. The more relevant information there is in the context, the less the model has to rely on its internal knowledge, which can be unreliable due to its training data and training methodology. Studies have shown that having access to relevant information in the context can help the model generate more detailed responses while reducing hallucinations (\nLewis et al.\n, 2020).\nFor example, given the query “Will Acme’s fancy-printer-A300 print 100pps?”, the model will be able to respond better if it’s given the specifications of fancy-printer-A300. (Thanks Chetan Tekur for the example.)\nContext construction for foundation models is equivalent to feature engineering for classical ML models. They serve the same purpose: giving the model the necessary information to process an input.\nIn-context learning, learning from the context, is a form of continual learning. It enables a model to incorporate new information continually to make decisions, preventing it from becoming outdated. For example, a model trained on last-week data won’t be able to answer questions about this week unless the new information is included in its context. By updating a model’s context with the latest information, e.g. fancy-printer-A300’s latest specifications, the model remains up-to-date and can respond to queries beyond its cut-off date.\nRAGs\nThe most well-known pattern for context construction is RAG, Retrieval-Augmented Generation. RAG consists of two components: a generator (e.g. a language model) and a retriever, which retrieves relevant information from external sources.\nRetrieval isn’t unique to RAGs. It’s the backbone of search engines, recommender systems, log analytics, etc. Many retrieval algorithms developed for traditional retrieval systems can be used for RAGs.\nExternal memory sources typically contain unstructured data, such as memos, contracts, news updates, etc. They can be collectively called\ndocuments\n. A document can be 10 tokens or 1 million tokens. Naively retrieving whole documents can cause your context to be arbitrarily long. RAG typically requires documents to be split into\nmanageable chunks\n, which can be determined from the model’s maximum context length and your application’s latency requirements. To learn more about chunking and the optimal chunk size, see\nPinecone\n,\nLangchain\n,\nLlamaindex\n, and\nGreg Kamradt\n’s tutorials.\nOnce data from external memory sources has been loaded and chunked, retrieval is performed using two main approaches.\nTerm-based retrieval\nThis can be as simple as keyword search. For example, given the query “transformer”, fetch all documents containing this keyword. More sophisticated algorithms include BM25 (which leverages TF-IDF) and Elasticsearch (which leverages inverted index).\nTerm-based retrieval is usually used for text data, but it also works for images and videos that have text metadata such as titles, tags, captions, comments, etc.\nEmbedding-based retrieval\n(also known as vector search)\nYou convert chunks of data into embedding vectors using an embedding model such as\nBERT\n,\nsentence-transformers\n, and proprietary embedding models provided by OpenAI or Google. Given a query, the data whose vectors are closest to the query embedding, as determined by the vector search algorithm, is retrieved.\nVector search is usually framed as nearest-neighbor search, using approximate nearest neighbor (ANN) algorithms such as\nFAISS\n(Facebook AI Similarity Search), Google’s\nScaNN\n, Spotify’s\nANNOY\n, and\nhnswlib\n(\nHierarchical Navigable Small World\n).\nThe\nANN-benchmarks website\ncompares different ANN algorithms on multiple datasets using four main metrics, taking into account the tradeoffs between indexing and querying.\nRecall\n: the fraction of the nearest neighbors found by the algorithm.\nQuery per second (QPS)\n: the number of queries the algorithm can handle per second. This is crucial for high-traffic applications.\nBuild time\n: the time required to build the index. This metric is important especially if you need to frequently update your index (e.g. because your data changes).\nIndex size\n: the size of the index created by the algorithm, which is crucial for assessing its scalability and storage requirements.\nThis works with not just text documents, but also images, videos, audio, and code. Many teams even try to summarize SQL tables and dataframes and then use these summaries to generate embeddings for retrieval.\nTerm-based retrieval is much faster and cheaper than embedding-based retrieval. It can work well out of the box, making it an attractive option to start. Both BM25 and Elasticsearch are widely used in the industry and serve as formidable baselines for more complex retrieval systems. Embedding-based retrieval, while computationally expensive, can be significantly improved over time to outperform term-based retrieval.\nA production retrieval system typically combines several approaches. Combining term-based retrieval and embedding-based retrieval is called\nhybrid search\n.\nOne common pattern is sequential. First, a cheap, less precise retriever, such as a term-based system, fetches candidates. Then, a more precise but more expensive mechanism, such as k-nearest neighbors, finds the best of these candidates. The second step is also called reranking.\nFor example, given the term “transformer”, you can fetch all documents that contain the word transformer, regardless of whether they are about the electric device, the neural architecture, or the movie. Then you use vector search to find among these documents those that are actually related to your transformer query.\nContext reranking differs from traditional search reranking in that the exact position of items is less critical. In search, the rank (e.g., first or fifth) is crucial. In context reranking, the order of documents still matters because it affects how well a model can process them. Models might better understand documents at the beginning and end of the context, as suggested by the paper\nLost in the middle\n(Liu et al., 2023). However, as long as a document is included, the impact of its order is less significant compared to in search ranking.\nAnother pattern is ensemble. Remember that a retriever works by ranking documents by their relevance scores to the query. You use multiple retrievers to fetch candidates at the same time, then combine these different rankings together to generate a final ranking.\nRAGs with tabular data\nExternal data sources can also be structured, such as dataframes or SQL tables. Retrieving data from an SQL table is significantly different from retrieving data from unstructured documents. Given a query, the system works as follows.\nText-to-SQL\n: Based on the user query and the table schemas, determine what SQL query is needed.\nSQL execution\n: Execute the SQL query.\nGeneration\n: Generate a response based on the SQL result and the original user query.\nFor the text-to-SQL step, if there are many available tables whose schemas can’t all fit into the model context, you might need an intermediate step to predict what tables to use for each query. Text-to-SQL can be done by the same model used to generate the final response or one of many specialized text-to-SQL models.\nAgentic RAGs\nAn important source of data is the Internet. A web search tool like Google or Bing API can give the model access to a rich, up-to-date resource to gather relevant information for each query. For example, given the query “Who won Oscar this year?”, the system searches for information about the latest Oscar and uses this information to generate the final response to the user.\nTerm-based retrieval, embedding-based retrieval, SQL execution, and web search are actions that a model can take to augment its context. You can think of each action as a function the model can call. A workflow that can incorporate external actions is also called\nagentic\n. The architecture then looks like this.\n» Action vs. tool «\nA tool allows one or more actions. For example, a people search tool might allow two actions: search by name and search by email. However, the difference is minimal, so many people use\naction\nand\ntool\ninterchangeably.\n» Read-only actions vs. write actions «\nActions that retrieve information from external sources but don’t change their states are read-only actions. Giving a model write actions, e.g. updating the values in a table, enables the model to perform more tasks but also poses more risks, which will be discussed later.\nQuery rewriting\nOften, a user query needs to be rewritten to increase the likelihood of fetching the right information. Consider the following conversation.\nUser: When was the last time John Doe bought something from us?\nAI: John last bought a Fruity Fedora hat from us two weeks ago, on January 3, 2030.\nUser: How about Emily Doe?\nThe last question, “How about Emily Doe?”, is ambiguous. If you use this query verbatim to retrieve documents, you’ll likely get irrelevant results. You need to rewrite this query to reflect what the user is actually asking. The new query should make sense on its own. The last question should be rewritten to “When was the last time Emily Doe bought something from us?”\nQuery rewriting is typically done using other AI models, using a prompt similar to “Given the following conversation, rewrite the last user input to reflect what the user is actually asking.”\nQuery rewriting can get complicated, especially if you need to do identity resolution or incorporate other knowledge. If the user asks “How about his wife?”, you will first need to query your database to find out who his wife is. If you don’t have this information, the rewriting model should acknowledge that this query isn’t solvable instead of hallucinating a name, leading to a wrong answer.\nStep 2. Put in Guardrails\nGuardrails help reduce AI risks and protect not just your users but also you, the developers. They should be placed whenever there is potential for failures. This post discusses two types of guardrails: input guardrails and output guardrails.\nInput guardrails\nInput guardrails are typically protection against two types of risks: leaking private information to external APIs, and executing bad prompts that compromise your system (model jailbreaking).\nLeaking private information to external APIs\nThis risk is specific to using external model APIs when you need to send your data outside your organization. For example, an employee might copy the company’s secret or a user’s private information into a prompt and send it to wherever the model is hosted.\nOne of the most notable early incidents was when Samsung employees put Samsung’s proprietary information into ChatGPT, accidentally\nleaking the company’s secrets\n. It’s unclear how Samsung discovered this leak and how the leaked information was used against Samsung. However, the incident was serious enough for Samsung to\nban ChatGPT in May 2023\n.\nThere’s no airtight way to eliminate potential leaks when using third-party APIs. However, you can mitigate them with guardrails. You can use one of the many available tools that automatically detect sensitive data. What sensitive data to detect is specified by you. Common sensitive data classes are:\nPersonal information (ID numbers, phone numbers, bank accounts).\nHuman faces.\nSpecific keywords and phrases associated with the company’s intellectual properties or privileged information.\nMany sensitive data detection tools use AI to identify potentially sensitive information, such as determining if a string resembles a valid home address. If a query is found to contain sensitive information, you have two options: block the entire query or remove the sensitive information from it. For instance, you can mask a user’s phone number with the placeholder [PHONE NUMBER]. If the generated response contains this placeholder, use a PII reversible dictionary that maps this placeholder to the original information so that you can unmask it, as shown below.\nModel jailbreaking\nIt’s become an online sport to try to jailbreak AI models, getting them to say or do bad things. While some might find it amusing to get ChatGPT to make controversial statements, it’s much less fun if your customer support chatbot, branded with your name and logo, does the same thing. This can be especially dangerous for AI systems that have access to tools. Imagine if a user finds a way to get your system to execute an SQL query that corrupts your data.\nTo combat this, you should first put guardrails on your system so that no harmful actions can be automatically executed. For example, no SQL queries that can insert, delete, or update data can be executed without human approval. The downside of this added security is that it can slow down your system.\nTo prevent your application from making outrageous statements it shouldn’t be making, you can define out-of-scope topics for your application. For example, if your application is a customer support chatbot, it shouldn’t answer political or social questions. A simple way to do so is to filter out inputs that contain predefined phrases typically associated with controversial topics, such as “immigration” or “antivax”. More sophisticated algorithms use AI to classify whether an input is about one of the pre-defined restricted topics.\nIf harmful prompts are rare in your system, you can use an anomaly detection algorithm to identify unusual prompts.\nOutput guardrails\nAI models are probabilistic, making their outputs unreliable. You can put in guardrails to significantly improve your application’s reliability. Output guardrails have two main functionalities:\nEvaluate the quality of each generation.\nSpecify the policy to deal with different failure modes.\nOutput quality measurement\nTo catch outputs that fail to meet your standards, you need to understand what failures look like. Here are examples of failure modes and how to catch them.\nEmpty responses\n.\nMalformatted responses\nthat don’t follow the expected output format. For example, if the application expects JSON and the generated response has a missing closing bracket. There are validators for certain formats, such as regex, JSON, and Python code validators. There are also tools for\nconstrained sampling\nsuch as guidance, outlines, and instructor.\nToxic responses\n, such as those that are racist or sexist. These responses can be caught using one of many toxicity detection tools.\nFactual inconsistent responses\nhallucinated by the model. Hallucination detection is an active area of research with solutions such as\nSelfCheckGPT\n(Manakul et al., 2023) and\nSAFE\n, Search Engine Factuality Evaluator (Wei et al., 2024). You can mitigate hallucinations by providing models with sufficient context and prompting techniques such as chain-of-thought. Hallucination detection and mitigation are discussed further in my upcoming book\nAI Engineering\n.\nResponses that contain sensitive information\n. This can happen in two scenarios.\nYour model was trained on sensitive data and regurgitates it back.\nYour system retrieves sensitive information from your internal database to enrich its context, and then it passes this sensitive information on to the response.\nThis failure mode can be prevented by not training your model on sensitive data and not allowing it to retrieve sensitive data in the first place. Sensitive data in outputs can be detected using the same tools used for input guardrails.\nBrand-risk responses\n, such as responses that mischaracterize your company or your competitors. An example is when Grok, a model trained by X, generated a response\nsuggesting that Grok was trained by OpenAI\n, causing the Internet to suspect X of stealing OpenAI’s data. This failure mode can be mitigated with keyword monitoring. Once you’ve identified outputs concerning your brands and competitors, you can either block these outputs, pass them onto human reviewers, or use other models to detect the sentiment of these outputs to ensure that only the right sentiments are returned.\nGenerally bad responses\n. For example, if you ask the model to write an essay and that essay is just bad, or if you ask the model for a low-calorie cake recipe and the generated recipe contains an excessive amount of sugar. It’s become a popular practice to use AI judges to evaluate the quality of models’ responses. These AI judges can be general-purpose models (think ChatGPT, Claude) or specialized scorers trained to output a concrete score for a response given a query.\nFailure management\nAI models are probabilistic, which means that if you try a query again, you might get a different response. Many failures can be mitigated using a basic retry logic. For example, if the response is empty, try again X times or until you get a non-empty response. Similarly, if the response is malformatted, try again until the model generates a correctly formatted response.\nThis retry policy, however, can incur extra latency and cost. One retry means 2x the number of API calls. If the retry is carried out after failure, the latency experienced by the user will double. To reduce latency, you can make calls in parallel. For example, for each query, instead of waiting for the first query to fail before retrying, you send this query to the model twice at the same time, get back two responses, and pick the better one. This increases the number of redundant API calls but keeps latency manageable.\nIt’s also common to fall back on humans to handle tricky queries. For example, you can transfer a query to human operators if it contains specific key phrases. Some teams use a specialized model, potentially trained in-house, to decide when to transfer a conversation to humans. One team, for instance, transfers a conversation to human operators when their sentiment analysis model detects that the user is getting angry. Another team transfers a conversation after a certain number of turns to prevent users from getting stuck in an infinite loop.\nGuardrail tradeoffs\nReliability vs. latency tradeoff\n: While acknowledging the importance of guardrails, some teams told me that latency is more important. They decided not to implement guardrails because they can significantly increase their application’s latency. However, these teams are in the minority. Most teams find that the increased risks are more costly than the added latency.\nOutput guardrails might not work well in the stream completion mode. By default, the whole response is generated before shown to the user, which can take a long time. In the stream completion mode, new tokens are streamed to the user as they are generated, reducing the time the user has to wait to see the response. The downside is that it’s hard to evaluate partial responses, so unsafe responses might be streamed to users before the system guardrails can determine that they should be blocked.\nSelf-hosted vs. third-party API tradeoff\n: Self-hosting your models means that you don’t have to send your data to a third party, reducing the need for input guardrails. However, it also means that you must implement all the necessary guardrails yourself, rather than relying on the guardrails provided by third-party services.\nOur platform now looks like this. Guardrails can be independent tools or parts of model gateways, as discussed later. Scorers, if used, are grouped under model APIs since scorers are typically AI models, too. Models used for scoring are typically smaller and faster than models used for generation.\nStep 3. Add Model Router and Gateway\nAs applications grow in complexity and involve more models, two types of tools emerged to help you work with multiple models: routers and gateways.\nRouter\nAn application can use different models to respond to different types of queries. Having different solutions for different queries has several benefits. First, this allows you to have specialized solutions, such as one model specialized in technical troubleshooting and another specialized in subscriptions. Specialized models can potentially perform better than a general-purpose model. Second, this can help you save costs. Instead of routing all queries to an expensive model, you can route simpler queries to cheaper models.\nA router typically consists of\nan intent classifier\nthat predicts what the user is trying to do. Based on the predicted intent, the query is routed to the appropriate solution. For example, for a customer support chatbot, if the intent is:\nTo reset a password –> route this user to the page about password resetting.\nTo correct a billing mistake –> route this user to a human operator.\nTo troubleshoot a technical issue –> route this query to a model finetuned for troubleshooting.\nAn intent classifier can also help your system avoid out-of-scope conversations. For example, you can have an intent classifier that predicts whether a query is out of the scope. If the query is deemed inappropriate (e.g. if the user asks who you would vote for in the upcoming election), the chatbot can politely decline to engage using one of the stock responses (“As a chatbot, I don’t have the ability to vote. If you have questions about our products, I’d be happy to help.”) without wasting an API call.\nIf your system has access to multiple actions, a router can involve a\nnext-action predictor\nto help the system decide what action to take next. One valid action is to ask for clarification if the query is ambiguous. For example, in response to the query “Freezing,” the system might ask, “Do you want to freeze your account or are you talking about the weather?” or simply say, “I’m sorry. Can you elaborate?”\nIntent classifiers and next-action predictors can be general-purpose models or specialized classification models. Specialized classification models are typically much smaller and faster than general-purpose models, allowing your system to use multiple of them without incurring significant extra latency and cost.\nWhen routing queries to models with varying context limits, the query’s context might need to be adjusted accordingly. Consider a query of 1,000 tokens that is slated for a model with a 4K context limit. The system then takes an action, e.g. web search, that brings back 8,000-token context. You can either truncate the query’s context to fit the originally intended model or route the query to a model with a larger context limit.\nGateway\nA model gateway is an intermediate layer that allows your organization to interface with different models in a unified and secure manner. The most basic functionality of a model gateway is to enable developers to access different models – be it self-hosted models or models behind commercial APIs such as OpenAI or Google – the same way. A model gateway makes it easier to maintain your code. If a model API changes, you only need to update the model gateway instead of having to update all applications that use this model API.\nIn its simplest form, a model gateway is a unified wrapper that looks like the following code example. This example is to give you an idea of how a model gateway might be implemented. It’s not meant to be functional as it doesn’t contain any error checking or optimization.\nimport google.generativeai as genai\nimport openai\n\ndef openai_model(input_data, model_name, max_tokens):\n    openai.api_key = os.environ[\"OPENAI_API_KEY\"]\n    response = openai.Completion.create(\n        engine=model_name,\n        prompt=input_data,\n        max_tokens=max_tokens\n    )\n    return {\"response\": response.choices[0].text.strip()}\n\ndef gemini_model(input_data, model_name, max_tokens):\n    genai.configure(api_key=os.environ[\"GOOGLE_API_KEY\"])\n    model = genai.GenerativeModel(model_name=model_name)\n    response = model.generate_content(input_data, max_tokens=max_tokens)\n    return {\"response\": response[\"choices\"][0][\"message\"][\"content\"]}\n\n@app.route('/model', methods=['POST'])\ndef model_gateway():\n    data = request.get_json()\n    model_type = data.get(\"model_type\")\n    model_name = data.get(\"model_name\")\n    input_data = data.get(\"input_data\")\n    max_tokens = data.get(\"max_tokens\")\n\n    if model_type == \"openai\":\n        result = openai_model(input_data, model_name, max_tokens)\n    elif model_type == \"gemini\":\n        result = gemini_model(input_data, model_name, max_tokens)\n    return jsonify(result)\nA model gateway is\naccess control and cost management\n. Instead of giving everyone who wants access to the OpenAI API your organizational tokens, which can be easily leaked, you only give people access to the model gateway, creating a centralized and controlled point of access. The gateway can also implement fine-grained access controls, specifying which user or application should have access to which model. Moreover, the gateway can monitor and limit the usage of API calls, preventing abuse and managing costs effectively.\nA model gateway can also be used to implement fallback policies to overcome rate limits or API failures (the latter is unfortunately common). When the primary API is unavailable, the gateway can route requests to alternative models, retry after a short wait, or handle failures in other graceful manners. This ensures that your application can operate smoothly without interruptions.\nSince requests and responses are already flowing through the gateway, it’s a good place to implement other functionalities such as load balancing, logging, and analytics. Some gateway services even provide caching and guardrails.\nGiven that gateways are relatively straightforward to implement, there are many off-the-shelf gateways. Examples include Portkey’s\ngateway\n,\nMLflow AI Gateway\n, WealthSimple’s\nllm-gateway\n,\nTrueFoundry\n,\nKong\n, and\nCloudflare\n.\nWith the added gateway and routers, our platform is getting more exciting. Like scoring, routing is also in the model gateway. Like models used for scoring, models used for routing are typically smaller than models used for generation.\nStep 4. Reduce Latency with Cache\nWhen I shared this post with my friend Eugene Yan, he said that cache is perhaps the most underrated component of an AI platform. Caching can significantly reduce your application’s latency and cost.\nCache techniques can also be used during training, but since this post is about deployment, I’ll focus on cache for inference. Some common inference caching techniques include prompt cache, exact cache, and semantic cache. Prompt cache are typically implemented by the inference APIs that you use. When evaluating an inference library, it’s helpful to understand what cache mechanism it supports.\nKV cache for the attention mechanism is out of scope for this discussion.\nPrompt cache\nMany prompts in an application have overlapping text segments. For example, all queries can share the same system prompt. A prompt cache stores these overlapping segments for reuse, so you only need to process them once. A common overlapping text segment in different prompts is the system prompt. Without prompt cache, your model needs to process the system prompt with every query. With prompt cache, it only needs to process the system prompt once for the first query.\nFor applications with long system prompts, prompt cache can significantly reduce both latency and cost. If your system prompt is 1000 tokens and your application generates 1 million model API calls today, a prompt cache will save you from processing approximately 1 billion repetitive input tokens a day! However, this isn’t entirely free. Like KV cache, prompt cache size can be quite large and require significant engineering effort.\nPrompt cache is also useful for queries that involve long documents. For example, if many of your user queries are related to the same long document (such as a book or a codebase), this long document can be cached for reuse across queries.\nSince its introduction in November 2023 by\nGim et al.\n, prompt cache has already been incorporated into model APIs. Google announced that Gemini APIs will offer this functionality in June 2024 under the name\ncontext cache\n. Cached input tokens are given a 75% discount compared to regular input tokens, but you’ll have to pay extra for cache storage (as of writing, $1.00 / 1 million tokens per hour). Given the obvious benefits of prompt cache, I wouldn’t be surprised if it becomes as popular as KV cache.\nWhile llama.cpp also has\nprompt cache\n, it seems to only cache whole prompts and work for queries in the same chat session. Its documentation is limited, but my guess from reading the code is that in a long conversation, it caches the previous messages and only processes the newest message.\nExact cache\nIf prompt cache and KV cache are unique to foundation models, exact cache is more general and straightforward. Your system stores processed items for reuse later when the exact items are requested. For example, if a user asks a model to summarize a product, the system checks the cache to see if a summary of this product is cached. If yes, fetch this summary. If not, summarize the product and cache the summary.\nExact cache is also used for embedding-based retrieval to avoid redundant vector search. If an incoming query is already in the vector search cache, fetch the cached search result. If not, perform a vector search for this query and cache the result.\nCache is especially appealing for queries that require multiple steps (e.g. chain-of-thought) and/or time-consuming actions (e.g. retrieval, SQL execution, or web search).\nAn exact cache can be implemented using in-memory storage for fast retrieval. However, since in-memory storage is limited, a cache can also be implemented using databases like PostgreSQL, Redis, or tiered storage to balance speed and storage capacity. Having an eviction policy is crucial to manage the cache size and maintain performance. Common eviction policies include Least Recently Used (LRU), Least Frequently Used (LFU), and First In, First Out (FIFO).\nHow long to cache a query depends on how likely this query is to be called again. User-specific queries such as “What’s the status of my recent order” are less likely to be reused by other users, and therefore, shouldn’t be cached. Similarly, it makes less sense to cache time-sensitive queries such as “How’s the weather?” Some teams train a small classifier to predict whether a query should be cached.\nSemantic cache\nUnlike exact cache, semantic cache doesn’t require the incoming query to be identical to any of the cached queries. Semantic cache allows the reuse of similar queries. Imagine one user asks “What’s the capital of Vietnam?” and the model generates the answer “Hanoi”. Later, another user asks “What’s the capital\ncity\nof Vietnam?”, which is the same question but with the extra word “city”. The idea of semantic cache is that the system can reuse the answer “Hanoi” instead of computing the new query from scratch.\nSemantic cache only works if you have a reliable way to determine if two queries are semantically similar. One common approach is embedding-based similarity, which works as follows:\nFor each query, generate its embedding using an embedding model.\nUse vector search to find the cached embedding closest to the current query embedding. Let’s say this similarity score is X.\nIf X is more than the similarity threshold you set, the cached query is considered the same as the current query, and the cached results are returned. If not, process this current query and cache it together with its embedding and results.\nThis approach requires a vector database to store the embeddings of cached queries.\nCompared to other caching techniques, semantic cache’s value is more dubious because many of its components are prone to failure.\nIts success relies on high-quality embeddings, functional vector search, and a trustworthy similarity metric. Setting the right similarity threshold can also be tricky and require a lot of trial and error. If the system mistakes the incoming query as being similar to another query, the returned response, fetched from the cache, will be incorrect.\nIn addition, semantic cache can be time-consuming and compute-intensive, as it involves a vector search. The speed and cost of this vector search depend on the size of your database of cached embeddings.\nSemantic cache might still be worth it if the cache hit rate is high, meaning that a good portion of queries can be effectively answered by leveraging the cached results. However, before incorporating the complexities of semantic cache, make sure to evaluate the efficiency, cost, and performance risks associated with it.\nWith the added cache systems, the platform looks as follows. KV cache and prompt cache are typically implemented by model API providers, so they aren’t shown in this image. If I must visualize them, I’d put them in the Model API box. There’s a new arrow to add generated responses to the cache.\nStep 5. Add Complex Logic and Write Actions\nThe applications we’ve discussed so far have fairly simple flows. The outputs generated by foundation models are mostly returned to users (unless they don’t pass the guardrails). However, an application flow can be more complex with loops and conditional branching. A model’s outputs can also be used to invoke write actions, such as composing an email or placing an order.\nComplex logic\nOutputs from a model can be conditionally passed onto another model or fed back to the same model as part of the input to the next step. This goes on until a model in the system decides that the task has been completed and that a final response should be returned to the user.\nThis can happen when you give your system the ability to plan and decide what to do next. As an example, consider the query “Plan a weekend itinerary for Paris.” The model might first generate a list of potential activities: visiting the Eiffel Tower, having lunch at a café, touring the Louvre, etc. Each of these activities can then be fed back into the model to generate more detailed plans. For instance, “visiting the Eiffel Tower” could prompt the model to generate sub-tasks like checking the opening hours, buying tickets, and finding nearby restaurants. This iterative process continues until a comprehensive and detailed itinerary is created.\nOur infrastructure now has an arrow pointing the generated response back to context construction, which in turn feeds back to models in the model gateway.\nWrite actions\nActions used for context construction are\nread-only actions\n. They allow a model to read from its data sources to gather context. But a system can also\nwrite actions\n, making changes to the data sources and the world. For example, if the model outputs: “send an email to X with the message Y”, the system will invoke the action\nsend_email(recipient=X, message=Y)\n.\nWrite actions make a system vastly more capable. They can enable you to automate the whole customer outreach workflow: researching potential customers, finding their contacts, drafting emails, sending first emails, reading responses, following up, extracting orders, updating your databases with new orders, etc.\nHowever, the prospect of giving AI the ability to automatically alter our lives is frightening. Just as you shouldn’t give an intern the authority to delete your production database, you shouldn’t allow an unreliable AI to initiate bank transfers. Trust in the system’s capabilities and its security measures is crucial. You need to ensure that the system is protected from bad actors who might try to manipulate it into performing harmful actions.\nAI systems are vulnerable to cyber attacks like other software systems, but they also have another weakness:\nprompt injection\n. Prompt injection happens when an attacker manipulates input prompts into a model to get it to express undesirable behaviors. You can think of prompt injection as social engineering done on AI instead of humans.\nA scenario that many companies fear is that they give an AI system access to their internal databases, and attackers trick this system into revealing private information from these databases. If the system has write access to these databases, attackers can trick the system into corrupting the data.\nAny organization that wants to leverage AI needs to take safety and security seriously. However, these risks don’t mean that AI systems should never be given the ability to act in the real world. AI systems can fail, but humans can fail too. If we can get people to trust a machine to take us up into space, I hope that one day, securities will be sufficient for us to trust autonomous AI systems.\nObservability\nWhile I have placed observability in its own section, it should be integrated into the platform from the beginning rather than added later as an afterthought. Observability is crucial for projects of all sizes, and its importance grows with the complexity of the system.\nThis section provides the least information compared to the others. It’s impossible to cover all the nuances of observability in a blog post. Therefore, I will only give a brief overview of the three pillars of monitoring: logs, traces, and metrics. I won’t go into specifics or cover user feedback, drift detection, and debugging.\nMetrics\nWhen discussing monitoring, most people think of metrics. What metrics to track depends on what you want to track about your system, which is application-specific. However, in general, there are two types of metrics you want to track: model metrics and system metrics.\nSystem metrics tell you the state of your overall system. Common metrics are throughput, memory usage, hardware utilization, and service availability/uptime. System metrics are common to all software engineering applications. In this post, I’ll focus on model metrics.\nModel metrics assess your model’s performance, such as accuracy, toxicity, and hallucination rate. Different steps in an application pipeline also have their own metrics. For example, in a RAG application, the retrieval quality is often evaluated using context relevance and context precision. A vector database can be evaluated by how much storage it needs to index the data and how long it takes to query the data\nThere are various ways a model’s output can fail. It’s crucial to identify these issues and develop metrics to monitor them. For example, you might want to track how often your model times out, returns empty responses or produces malformatted responses. If you’re worried about your model revealing sensitive information, find a way to track that too.\nLength-related metrics such as query, context, and response length are helpful for understanding your model’s behaviors. Is one model more verbose than another? Are certain types of queries more likely to result in lengthy answers? They are especially useful for detecting changes in your application. If the average query length suddenly decreases, it could indicate an underlying issue that needs investigation.\nLength-related metrics are also important for tracking latency and costs, as longer contexts and responses typically increase latency and incur higher costs.\nTracking latency is essential for understanding the user experience. Common latency metrics include:\nTime to First Token\n(TTFT): The time it takes for the first token to be generated.\nTime Between Tokens\n(TBT): The interval between each token generation.\nTokens Per Second\n(TPS): The rate at which tokens are generated.\nTime Per Output Token\n(TPOT): The time it takes to generate each output token.\nTotal Latency\n: The total time required to complete a response.\nYou’ll also want to track costs. Cost-related metrics are the number of queries and the volume of input and output tokens. If you use an API with rate limits, tracking the number of requests per second is important to ensure you stay within your allocated limits and avoid potential service interruptions.\nWhen calculating metrics, you can choose between spot checks and exhaustive checks. Spot checks involve sampling a subset of data to quickly identify issues, while exhaustive checks evaluate every request for a comprehensive performance view. The choice depends on your system’s requirements and available resources, with a combination of both providing a balanced monitoring strategy.\nWhen computing metrics, ensure they can be broken down by relevant axes, such as users, releases, prompt/chain versions, prompt/chain types, and time. This granularity helps in understanding performance variations and identifying specific issues.\nLogs\nSince this blog post is getting long and I’ve written at length about\nlogs\nin\nDesigning Machine Learning Systems\n, I will be quick here. The philosophy for logging is simple: log everything. Log the system configurations. Log the query, the output, and the intermediate outputs. Log when a component starts, ends, when something crashes, etc. When recording a piece of log, make sure to give it tags and IDs that can help you know where in the system this log comes from.\nLogging everything means that the amount of logs you have can grow very quickly. Many tools for automated log analysis and log anomaly detection are powered by AI.\nWhile it’s impossible to manually process logs, it’s useful to manually inspect your production data daily to get a sense of how users are using your application.\nShankar et al.\n(2024) found that the developers’ perceptions of what constitutes good and bad outputs change as they interact with more data, allowing them to both rewrite their prompts to increase the chance of good responses and update their evaluation pipeline to catch bad responses.\nTraces\nTrace refers to the detailed recording of a request’s execution path through various system components and services. In an AI application, tracing reveals the entire process from when a user sends a query to when the final response is returned, including the actions the system takes, the documents retrieved, and the final prompt sent to the model. It should also show how much time each step takes and its associated cost, if measurable. As an example, this is a visualization of a\nLangsmith\ntrace.\nIdeally, you should be able to trace each query’s transformation through the system step-by-step. If a query fails, you should be able to pinpoint the exact step where it went wrong: whether it was incorrectly processed, the retrieved context was irrelevant, or the model generated a wrong response.\nAI Pipeline Orchestration\nAn AI application can get fairly complex, consisting of multiple models, retrieving data from many databases, and having access to a wide range of tools. An orchestrator helps you specify how these different components are combined (chained) together to create an end-to-end application flow.\nAt a high level, an orchestrator works in two steps: components definition and chaining (also known as pipelining).\nComponents Definition\nYou need to tell the orchestrator what components your system uses, such as models (including models for generation, routing, and scoring), databases from which your system can retrieve data, and actions that your system can take. Direct integration with model gateways can help simplify model onboarding, and some orchestrator tools want to be gateways. Many orchestrators also support integration with tools for evaluation and monitoring.\nChaining (or pipelining)\nYou tell the orchestrator the sequence of steps your system takes from receiving the user query until completing the task. In short, chaining is just function composition. Here’s an example of what a pipeline looks like.\nProcess the raw query.\nRetrieve the relevant data based on the processed query.\nThe original query and the retrieved data are combined to create a prompt in the format expected by the model.\nThe model generates a response based on the prompt.\nEvaluate the response.\nIf the response is considered good, return it to the user. If not, route the query to a human operator.\nThe orchestrator is responsible for passing data between steps and can provide toolings that help ensure that the output from the current step is in the format expected by the next step.\nWhen designing the pipeline for an application with strict latency requirements, try to do as much in parallel as possible. For example, if you have a routing component (deciding where to send a query to) and a PII removal component, they can do both at the same time.\nThere are many AI orchestration tools, including\nLangChain\n,\nLlamaIndex\n,\nFlowise\n,\nLangflow\n, and\nHaystack\n. Each tool has its own APIs so I won’t show the actual code here.\nWhile it’s tempting to jump straight to an orchestration tool when starting a project, start building your application without one first.\nAny external tool brings added complexity. An orchestrator can abstract away critical details of how your system works, making it hard to understand and debug your system.\nAs you advance to the later stages of your application development process, you might decide that an orchestrator can make your life easier. Here are three aspects to keep in mind when evaluating orchestrators.\nIntegration and extensibility\nEvaluate whether the orchestrator supports the components you’re already using or might adopt in the future. For example, if you want to use a Llama model, check if the orchestrator supports that. Given how many models, databases, and frameworks there are, it’s impossible for an orchestrator to support everything. Therefore, you’ll also need to consider an orchestrator’s extensibility. If it doesn’t support a specific component, how hard it is to change that?\nSupport for complex pipelines\nAs your applications grow in complexity, you might need to manage intricate pipelines involving multiple steps and conditional logic. An orchestrator that supports advanced features like branching, parallel processing, and error handling will help you manage these complexities efficiently.\nEase of use, performance, and scalability\nConsider the user-friendliness of the orchestrator. Look for intuitive APIs, comprehensive documentation, and strong community support, as these can significantly reduce the learning curve for you and your team. Avoid orchestrators that initiate hidden API calls or introduce latency to your applications. Additionally, ensure that the orchestrator can scale effectively as the number of applications, developers, and traffic grows.\nConclusion\nThis post started with a basic architecture and then gradually added components to address the growing application complexities. Each addition brings its own set of benefits and challenges, requiring careful consideration and implementation.\nWhile the separation of components is important to keep your system modular and maintainable, this separation is fluid. There are many overlaps between components. For example, a model gateway can share functionalities with guardrails. Cache can be implemented in different components, such as in vector search and inference services.\nThis post is much longer than I intended it to be, and yet there are many details I haven’t been able to explore further, especially around observability, context construction, complex logic, cache, and guardrails. I’ll dive deeper into all these components in my upcoming book\nAI Engineering\n.\nThis post also didn’t discuss how to serve models, assuming that most people will be using models provided by third-party APIs. AI Engineering will also have a chapter dedicated to inference and model optimization.\nReferences and Acknowledgments\nSpecial thanks to\nLuke Metz\n,\nAlex Li\n,\nChetan Tekur\n,\nKittipat “Bot” Kampa\n,\nHien Luu\n, and\nDenys Linkov\nfor feedback on the early versions of this post. Their insights greatly improved the content. Any remaining errors are my own.\nI read many case studies shared by companies on how they adopted generative AI, and here are some of my favorites.\nMusings on Building a Generative AI Product\n(LinkedIn, 2024)\nHow we built Text-to-SQL at Pinterest\n(Pinterest, 2024)\nFrom idea to reality: Elevating our customer support through generative AI\n(Vimeo, 2023)\nA deep dive into the world’s smartest email AI\n(Shortwave, 2023)\nLLM-powered data classification for data entities at scale\n(Grab, 2023)\nFrom Predictive to Generative - How Michelangelo Accelerates Uber’s AI Journey\n(Uber, 2024)",
  "metadata": {
    "title": "Building A Generative AI Platform",
    "description": "After studying how companies deploy generative AI applications, I noticed many similarities in their platforms. This post outlines the common components of a...",
    "keywords": null,
    "author": "Chip Huyen",
    "og:title": "Building A Generative AI Platform",
    "og:locale": "en_US",
    "og:description": "After studying how companies deploy generative AI applications, I noticed many similarities in their platforms. This post outlines the common components of a generative AI platform, what they do, and how they are implemented. I try my best to keep the architecture general, but certain applications might deviate. This is what the overall architecture looks like.",
    "og:url": "https://huyenchip.com/2024/07/25/genai-platform.html",
    "og:site_name": "Chip Huyen",
    "og:image": "https://huyenchip.com/assets/pics/genai-platform/1-genai-platform.png",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "article:published_time": "2024-07-25T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/2baeb4feb113/code/body-000.txt",
      "chars": 175,
      "language": "text"
    },
    {
      "file": "attachments/2baeb4feb113/code/body-001.txt",
      "chars": 1132,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/1-genai-platform.png",
      "bytes": 208471,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-a18720434e7d2aba46b3.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/1-genai-platform.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/2.png",
      "bytes": 29961,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-d5c304b7c9843df28307.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/2.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/3-rag.png",
      "bytes": 75105,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-13d557fdb994f6c73d37.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/3-rag.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/4-rag-with-tabular-data.png",
      "bytes": 84277,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-44096608eaeeee1a3c21.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/4-rag-with-tabular-data.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/5-agentic-rag.png",
      "bytes": 122186,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-654cb09b884b2cfeddd1.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/5-agentic-rag.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/6-query-rewriting.png",
      "bytes": 97719,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-4d48d6fc1060ee08280d.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/6-query-rewriting.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/7-reversible-pii-mapping.png",
      "bytes": 344062,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-93aae6b13ab35ba52c6c.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/7-reversible-pii-mapping.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/8-guardrails.png",
      "bytes": 158416,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-2fc655af7920ee0067e9.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/8-guardrails.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/9-llm-gateway.png",
      "bytes": 87321,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-bb528ec2c312335e6714.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/9-llm-gateway.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/10-model-gateway.png",
      "bytes": 173570,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-dca58ccdf7f9ff8e7571.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/10-model-gateway.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/12-cache.png",
      "bytes": 184918,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-7adbca5aa0a19bb36c3c.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/12-cache.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/13-loop.png",
      "bytes": 186988,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-1b4b26d54fe9d11932e8.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/13-loop.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/14-write-actions.png",
      "bytes": 207294,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-b6d621a8e301dab3d7dc.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/14-write-actions.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/genai-platform/15-traces.png",
      "bytes": 648499,
      "reused": false,
      "local_path": "attachments/2baeb4feb113/images/body-07f6ec349aecba21500d.png",
      "src": "https://huyenchip.com/assets/pics/genai-platform/15-traces.png",
      "alt": "Overview of a genai platform",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=8OJC21T2SL4",
      "src": "https://www.youtube.com/watch?v=8OJC21T2SL4",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/2baeb4feb113/videos/repair-8OJC21T2SL4.mp4",
      "bytes": 264532914,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "4139.341000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://oreillymedia.pxf.io/GmaeBn",
      "label": "AI Engineering",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.11401",
      "label": "Lewis et al.",
      "type": "a"
    },
    {
      "href": "https://www.pinecone.io/learn/chunking-strategies/",
      "label": "Pinecone",
      "type": "a"
    },
    {
      "href": "https://js.langchain.com/v0.1/docs/modules/data_connection/document_transformers/",
      "label": "Langchain",
      "type": "a"
    },
    {
      "href": "https://docs.llamaindex.ai/en/stable/optimizing/production_rag/",
      "label": "Llamaindex",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=8OJC21T2SL4",
      "label": "Greg Kamradt",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1810.04805",
      "label": "BERT",
      "type": "a"
    },
    {
      "href": "https://github.com/UKPLab/sentence-transformers",
      "label": "sentence-transformers",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1702.08734",
      "label": "FAISS",
      "type": "a"
    },
    {
      "href": "https://research.google/blog/announcing-scann-efficient-vector-similarity-search/",
      "label": "ScaNN",
      "type": "a"
    },
    {
      "href": "https://github.com/spotify/annoy",
      "label": "ANNOY",
      "type": "a"
    },
    {
      "href": "https://github.com/nmslib/hnswlib",
      "label": "hnswlib",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1603.09320",
      "label": "Hierarchical Navigable Small World",
      "type": "a"
    },
    {
      "href": "https://ann-benchmarks.com/",
      "label": "ANN-benchmarks website",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2307.03172",
      "label": "Lost in the middle",
      "type": "a"
    },
    {
      "href": "https://www.techradar.com/news/samsung-workers-leaked-company-secrets-by-using-chatgpt",
      "label": "leaking the company’s secrets",
      "type": "a"
    },
    {
      "href": "https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-other-generative-ai-use-by-staff-after-leak",
      "label": "ban ChatGPT in May 2023",
      "type": "a"
    },
    {
      "href": "https://huyenchip.com/2024/01/16/sampling.html",
      "label": "constrained sampling",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.08896",
      "label": "SelfCheckGPT",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2403.18802",
      "label": "SAFE",
      "type": "a"
    },
    {
      "href": "https://learning.oreilly.com/library/view/ai-engineering/9781098166298/",
      "label": "AI Engineering",
      "type": "a"
    },
    {
      "href": "https://x.com/JaxWinterbourne/status/1733339886155968714",
      "label": "suggesting that Grok was trained by OpenAI",
      "type": "a"
    },
    {
      "href": "https://github.com/Portkey-AI/gateway",
      "label": "gateway",
      "type": "a"
    },
    {
      "href": "https://mlflow.org/docs/latest/llms/gateway/index.html",
      "label": "MLflow AI Gateway",
      "type": "a"
    },
    {
      "href": "https://github.com/wealthsimple/llm-gateway",
      "label": "llm-gateway",
      "type": "a"
    },
    {
      "href": "https://docs.truefoundry.com/docs/ai-gateway",
      "label": "TrueFoundry",
      "type": "a"
    },
    {
      "href": "https://konghq.com/products/kong-ai-gateway",
      "label": "Kong",
      "type": "a"
    },
    {
      "href": "https://developers.cloudflare.com/ai-gateway/",
      "label": "Cloudflare",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2311.04934",
      "label": "Gim et al.",
      "type": "a"
    },
    {
      "href": "https://ai.google.dev/gemini-api/docs/caching",
      "label": "context cache",
      "type": "a"
    },
    {
      "href": "https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md",
      "label": "prompt cache",
      "type": "a"
    },
    {
      "href": "https://learning.oreilly.com/library/view/designing-machine-learning/9781098107956/ch08.html",
      "label": "logs",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969?&_encoding=UTF8&tag=chiphuyen-20&linkCode=ur2&linkId=0a1dbab0e76f5996e29e1a97d45f14a5&camp=1789&creative=9325",
      "label": "Designing Machine Learning Systems",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2404.12272",
      "label": "Shankar et al.",
      "type": "a"
    },
    {
      "href": "https://blog.langchain.dev/announcing-langsmith/",
      "label": "Langsmith",
      "type": "a"
    },
    {
      "href": "https://github.com/langchain-ai/langchain",
      "label": "LangChain",
      "type": "a"
    },
    {
      "href": "https://github.com/run-llama/llama_index",
      "label": "LlamaIndex",
      "type": "a"
    },
    {
      "href": "https://github.com/FlowiseAI/Flowise",
      "label": "Flowise",
      "type": "a"
    },
    {
      "href": "https://github.com/langflow-ai/langflow",
      "label": "Langflow",
      "type": "a"
    },
    {
      "href": "https://github.com/deepset-ai/haystack",
      "label": "Haystack",
      "type": "a"
    },
    {
      "href": "https://x.com/luke_metz",
      "label": "Luke Metz",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/findalexli/",
      "label": "Alex Li",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/chetantekur/",
      "label": "Chetan Tekur",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/kittipat-bot-kampa-1b1965/",
      "label": "Kittipat “Bot” Kampa",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/hienluu/",
      "label": "Hien Luu",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/denyslinkov/",
      "label": "Denys Linkov",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product?_l=en_US",
      "label": "Musings on Building a Generative AI Product",
      "type": "a"
    },
    {
      "href": "https://medium.com/pinterest-engineering/how-we-built-text-to-sql-at-pinterest-30bad30dabff",
      "label": "How we built Text-to-SQL at Pinterest",
      "type": "a"
    },
    {
      "href": "https://medium.com/vimeo-engineering-blog/from-idea-to-reality-elevating-our-customer-support-through-generative-ai-101a2c5ea680",
      "label": "From idea to reality: Elevating our customer support through generative AI",
      "type": "a"
    },
    {
      "href": "https://www.shortwave.com/blog/deep-dive-into-worlds-smartest-email-ai/",
      "label": "A deep dive into the world’s smartest email AI",
      "type": "a"
    },
    {
      "href": "https://engineering.grab.com/llm-powered-data-classification",
      "label": "LLM-powered data classification for data entities at scale",
      "type": "a"
    },
    {
      "href": "https://www.uber.com/blog/from-predictive-to-generative-ai/",
      "label": "From Predictive to Generative - How Michelangelo Accelerates Uber’s AI Journey",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:48Z",
  "source_url": "https://huyenchip.com/2024/07/25/genai-platform.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.11401",
      "bytes": 885323,
      "reused": true,
      "local_path": "papers/repair/fc8d32f196e0f0c27ffb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1810.04805",
      "bytes": 775166,
      "reused": true,
      "local_path": "papers/repair/33769e0bafc9f0c73baa.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1702.08734",
      "bytes": 1233821,
      "reused": false,
      "local_path": "papers/repair/8e3bbab15efefc1a5488.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1603.09320",
      "bytes": 2636958,
      "reused": true,
      "local_path": "papers/repair/7f2db157afe85482dc33.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2307.03172",
      "bytes": 747542,
      "reused": false,
      "local_path": "papers/repair/9a55f69abe89ff9892a1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.08896",
      "bytes": 903117,
      "reused": true,
      "local_path": "papers/repair/6203039f8fe36d722964.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2403.18802",
      "bytes": 983383,
      "reused": true,
      "local_path": "papers/repair/f8d6b7bcdeff30e3d5db.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2311.04934",
      "bytes": 905557,
      "reused": false,
      "local_path": "papers/repair/b8fcc2224156a0ae4431.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2404.12272",
      "bytes": 2613163,
      "reused": false,
      "local_path": "papers/repair/1db96d99ad72b559bec0.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 50585,
  "expected_body_images": 14,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/2baeb4feb113/pages/source.html",
  "local_html": "attachments/2baeb4feb113/pages/article.html",
  "body_base_path": "attachments/2baeb4feb113/pages/",
  "body_markdown_path": "attachments/2baeb4feb113/pages/article.md"
}