{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Eugene Yan",
  "title_listed": "How to Match LLM Patterns to Problems",
  "keywords": "LLM,模式匹配,选型",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://eugeneyan.com/writing/llm-problems/",
  "publish_date": "2023-07",
  "corpus_stage": "推理部署",
  "notes": "LLM模式与问题的匹配方法",
  "id": "6c481b82cf88",
  "collected_at": "2026-10-02T14:16:48",
  "title": "How to Match LLM Patterns to Problems",
  "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nHow to Match LLM Patterns to Problems\n[\nllm\nproduction\n]\n· 6 min read\nAfter my previous write-up on\nLLM patterns\n, I’ve received questions on how to match those patterns to various LLM problems. Thus, in this follow-up, we’ll discuss some potential problems faced when using LLMs and the patterns that help mitigate them.\nExternal vs. internal LLMs, data vs. non-data patterns\nBefore we dive into it, I think it’s helpful to distinguish between external vs. internal LLMs.\nExternal LLMs are models we don’t have full control over.\nWe can’t fine-tune them, are constrained by rate/token limits, and may have concerns with sending them confidential or proprietary data. Nonetheless, they’re currently the SOTA in terms of quality and safety.\nInternal LLMs are those we develop and host ourselves\n. While they may not have the constraints of external LLMs, we incur the cost of developing and hosting these LLMs. Also, we typically build them on top of open LLMs (unless you have a couple million to pre-train your own). Unfortunately, these open LLMs tend to be months, if not years, behind the best 3rd-party provider LLMs.\nWith regard to patterns, I think it’s useful to understand the role data plays: Is data the main component or a by-product? Or is the pattern minimally related to data?\nFor example,\nevals and fine-tuning are tied to data\n. We can’t do evals without gold labels or references; we can’t fine-tune without demonstration data. In contrast, patterns such as\ncaching, defensive UX, and guardrails have more to do with infra and UI than data\n.\nOn the other hand, patterns such as\nRAG and collecting user feedback lie somewhere in the middle.\nRAG provides data for in-context learning but needs to be backed by retrieval or search indices. User feedback helps with fine-tuning but needs to be collected via the right UX and refined via analysis and data pipelines.\nMatching patterns to problems\nHere are some LLM problems and the patterns that help address them. Since we’ve\npreviously discussed the patterns extensively\n, we’ll focus on the problems here.\nLack of performance metrics for our specific task.\nWhen benchmarking external and internal LLMs, we often want to—rightly or wrongly—summarize performance into a table of numbers. (Note: This is fiendishly difficult, especially for more abstract tasks where there are innumerable good outputs.) And as we tweak our systems via updating prompt templates, fine-tuning models, or improving RAG, we need a way to measure improvements or regressions. Finally, how do we measure if users like or dislike these new LLM-based features, and the impact of our tweaks?\nEvals\n: To benchmark across models and measure performance with each change\nCollect user feedback\n: To understand what users like or dislike\nExternal model performing poorly.\nThis can be due to the model not being trained on recent data (e.g., ChatGPT’s knowledge cutoff of Sept 2021) or proprietary data within your org (e.g., internal code and documents). Other concerns include hallucinations or incorrect responses because the model lacks sufficient or the most recent context.\nRAG\n: To provide relevant context, reducing hallucination and improving responses\nEvals\n: To measure performance improvements while tuning retrieval indices\nInternal model performing poorly.\nOpen LLMs tend to perform poorly on specific tasks in our specific domain out of the box. This can lead to defects such as poor extraction or summarization, non-factual responses, going off-topic, or simply lack of fluency. These LLMS may also return harmful responses that we want to minimize. While the patterns that apply to external LLMs are also relevant, because it’s an internal model, we have a few more tricks at our disposal.\nFine-tuning\n: To improve performance on specific tasks in our domain\nCollect user feedback\n: To use as user preference data for fine-tuning\nConstraints on external models.\nThese constraints can be technical (e.g., rate limits, latency, models being excessively fine-tuned), legal (e.g., not sending confidential or user private data, copyright over and use of external LLM output), and financial (i.e., high cost of API calls). AFAIK, there are two viable solutions: Either negotiate a contract with your external LLM provider or develop and self-host your own LLMs. I suggest saintly patience for the former and the patterns below for the latter.\nFine-tuning\n: To improve performance of internal models on our specific tasks\nEvals\n: To track progress of internal LLMs and compare them against external LLMs\nCollect user feedback\n: For fine-tuning and evaluation data\nLatency exceeds UX requirements\n. Certain use cases require the entire LLM output to be available within a few hundred milliseconds, including running guardrails on the output. While streaming output helps a ton with the UX, it may not be viable for certain user experiences and interfaces (no, I’m not referring to chat).\nCaching\n: Figuring out smart ways to generate (either in batch or asynchronously) and cache responses\nother\nthan via semantic similarity (e.g., item IDs)\nUnreliable or unusable model output.\nI categorize these as syntactic vs. semantic errors. Syntactic errors occur when the model doesn’t adhere to a specific format such as JSON or a specific sentence structure, or the generated code or SQL doesn’t run. Semantic errors occur when the model output is harmful, non-factual, off-topic, or simply incoherent.\nGuardrails\n(guidance + syntax checks): Guide LLM outputs; check for syntax errors\nGuardrails\n(semantic checks): Check for content safety, factuality, on-topic, etc.\nCustomer experience paper cuts.\nMachine learning models aren’t perfect—they will produce inaccurate output. The same goes for LLMs. Thus, after we’ve built a new LLM product/feature, how do we make it easy for users to explore and increase adoption? Also, how do we acknowledge that errors will happen, mitigate them, and earn trust over time?\nDefensive UX\n(for onboarding): Anchor on familiarity and set the right expectations\nDefensive UX\n(for paper cuts): Set the right expectations, support efficient dismissal and correction, and provide suitable attribution where available\nCollect user feedback\n: To understand which features work and don’t work, and to collect data for fine-tuning and evals to improve the next iteration\nLack of visibility on customer impact.\nHow do we know if our models are helping or hurting? Someone shared an anecdote of running an LLM-based customer support solution in prod for two weeks before discontinuing it—an A/B test showed that losses were 12x more when using an LLM as a substitute for their support team!\nMonitoring\n: Track metrics such as feature usage, user opt-outs, daily users, etc.\nCollect user feedback\n: Make it easy for customers to provide positive or negative feedback, and also analyze implicit feedback as a proxy for user engagement\n• • •\nAre there any key problems I’ve missed?\nPlease\nlet me know!\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Aug 2023). How to Match LLM Patterns to Problems. eugeneyan.com.\n        https://eugeneyan.com/writing/llm-problems/.\nor\n@article{yan2023llm-problems,\n  title   = {How to Match LLM Patterns to Problems},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2023},\n  month   = {Aug},\n  url     = {https://eugeneyan.com/writing/llm-problems/}\n}\nShare on:\nBrowse related tags:\n[\nllm\nproduction\n]\nor\nSearch\n« Patterns for Building LLM-based Systems & Products\nEvaluation & Hallucination Detection for Abstractive Summaries »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates",
  "metadata": {
    "title": "How to Match LLM Patterns to Problems",
    "description": "Distinguishing problems with external vs. internal LLMs, and data vs non-data patterns",
    "keywords": null,
    "author": "Eugene Yan",
    "og:site_name": "eugeneyan.com",
    "og:title": "How to Match LLM Patterns to Problems",
    "og:type": "article",
    "og:description": "Distinguishing problems with external vs. internal LLMs, and data vs non-data patterns",
    "og:url": "https://eugeneyan.com/writing/llm-problems/",
    "og:image": "https://eugeneyan.com/assets/og_image/finding-patterns.jpg",
    "twitter:card": "summary_large_image",
    "twitter:domain": "eugeneyan.com",
    "twitter:title": "How to Match LLM Patterns to Problems",
    "twitter:description": "Distinguishing problems with external vs. internal LLMs, and data vs non-data patterns",
    "twitter:image": "https://eugeneyan.com/assets/og_image/finding-patterns.jpg",
    "article:published_time": "2023-08-13T00:00:00+00:00",
    "article:author": "https://eugeneyan.com/about/",
    "article:section": "posts",
    "article:tag": "production"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/6c481b82cf88/code/body-000.txt",
      "chars": 236,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-search.svg",
      "bytes": 425,
      "reused": false,
      "local_path": "attachments/6c481b82cf88/images/body-feeb4bd87b8704a903b2.svg",
      "src": "https://eugeneyan.com/assets/icon-search.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-twitter.svg",
      "bytes": 892,
      "reused": false,
      "local_path": "attachments/6c481b82cf88/images/body-0083a7581b07e5f40e7b.svg",
      "src": "https://eugeneyan.com/assets/icon-twitter.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "bytes": 559,
      "reused": false,
      "local_path": "attachments/6c481b82cf88/images/body-f7a16c3e46e5838cec4d.svg",
      "src": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bluesky.svg",
      "bytes": 745,
      "reused": false,
      "local_path": "attachments/6c481b82cf88/images/body-be754e8ecdaf6de94f55.svg",
      "src": "https://eugeneyan.com/assets/bluesky.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-facebook.svg",
      "bytes": 1123,
      "reused": false,
      "local_path": "attachments/6c481b82cf88/images/body-5d55b59733923cf14223.svg",
      "src": "https://eugeneyan.com/assets/icon-facebook.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-mail.svg",
      "bytes": 841,
      "reused": false,
      "local_path": "attachments/6c481b82cf88/images/body-b179147617228dfafb9f.svg",
      "src": "https://eugeneyan.com/assets/icon-mail.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://eugeneyan.com/",
      "label": "eugeneyan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/start-here/",
      "label": "Start Here",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/",
      "label": "Writing",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/",
      "label": "Speaking",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/prototyping/",
      "label": "Prototyping",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/search/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/llm/",
      "label": "llm",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/production/",
      "label": "production",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/llm-patterns/",
      "label": "LLM patterns",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/practical-guide-to-maintaining-machine-learning/",
      "label": "Monitoring",
      "type": "a"
    },
    {
      "href": "https://twitter.com/eugeneyan",
      "label": "let me know!",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/abstractive/",
      "label": "Evaluation & Hallucination Detection for Abstractive Summaries »",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:15Z",
  "source_url": "https://eugeneyan.com/writing/llm-problems/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7727,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/6c481b82cf88/pages/source.html",
  "local_html": "attachments/6c481b82cf88/pages/article.html",
  "body_base_path": "attachments/6c481b82cf88/pages/",
  "body_markdown_path": "attachments/6c481b82cf88/pages/article.md"
}