{
  "source_file": "0930",
  "source_platform": "Ghost",
  "author": "Avi Chawla / Daily Dose of Data Science",
  "title_listed": "How To Reduce LLM Costs By 50 60 Using Model Routing",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.dailydoseofds.com/p/how-to-reduce-llm-costs-by-50-60-using-model-routing/",
  "publish_date": "2026-07-12",
  "corpus_stage": "",
  "notes": "",
  "id": "3b30edc9dac0",
  "collected_at": "2026-10-02T13:46:39",
  "title": "How to Reduce LLM Costs by 50-60% Using Model Routing",
  "body_markdown": "Jul 6, 2026\nAgents\nHow to Reduce LLM Costs by 50-60% Using Model Routing\n...explained with code.\nAvi Chawla\n👉\nBrian Armstrong from Coinbase said humans shouldn’t be the ones choosing which model answers a request. Model selection is a job for AI.\nCoinbase is already running on that belief. They built a routing layer in front of every request.\nLLM routing sits between the app and model providers, reads each prompt, classifies the task, and forwards it to the right model automatically.\nCoinbase used it to match each prompt to the right model and default to cheaper options where the task allowed it. Cache hit rate went from 5% to 60% and spend ended up nearly cut in half while usage kept growing.\nPlano (\nhere’s the open-source GitHub repo\n) is an open-source implementation of exactly that routing layer, the one that decides which model answers which request, without your application code ever specifying one.\nPlano GitHub Repo\nThe way it works is that you define routing preferences in plain English, and Plano orchestrator infers the domain and action of each incoming prompt, matches it to the right preference, and picks the model automatically.\nNext, let’s set up Plano in front of Hermes Agent, and watch it route requests to different models based on what each prompt actually needs.\nBefore the setup, let’s understand one key concept.\nThe problem with routing\nWhen an agent runs a task, it doesn’t send a single request. One user prompt triggers several LLM calls in sequence through tool use, each building on what came before.\nWith a hardcoded model, that’s never a problem. But once routing is in place, each of those calls gets classified independently.\nConsider an agent debugging a function. It makes a planning call, runs a tool, reads the output, then makes another call to analyze what came back.\nCall 1 routes to claude-sonnet-4.5 for code generation. Call 4 gets classified as analysis and routes to gpt-4o-mini instead.\nTwo problems occur when that happens.\nThe model on call 4 has no memory of decisions made in call 1.\nAnd every model switch invalidates the cache, so you’re paying full token rate again from scratch.\nPlano solves this with model affinity.\nThe first call in a session routes normally, and Plano pins that model to a session ID passed in a header.\nEvery subsequent call with the same ID goes to the same model regardless of how it gets classified.\nimport uuid\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:12000/v1\", api_key=\"EMPTY\")\naffinity_id = str(uuid.uuid4())\n\nresponse = client.chat.completions.create(\n    messages=messages,\n    tools=tools,\n    extra_headers={\"X-Model-Affinity\": affinity_id}\n)\nThe pin lasts 10 minutes by default.\nWhen the task changes, you generate a new ID, and routing starts fresh. For multi-replica deployments, backing it with Redis keeps the pin shared across instances:\nrouting:\n  session_ttl_seconds: 600\n  session_cache:\n    type: redis\n    url: redis://localhost:6379\nModel affinity handles consistency within a task. Now let’s build the routing layer using Plano.\nSetup\nPlano runs as a local proxy on your local machine. The routing config, model providers, and listener settings…everything is defined in one YAML file.\nPlano supports three routing methods:\nModel-based: the client specifies an exact model name on every call.\nAlias-based: you define a friendly name like\nfast-model\nor\nreasoning-model\nthat maps to a real model behind it, so your app never hardcodes a provider, and swapping models is one line in the config.\nPreference-aligned: the client specifies no model at all. Plano-Orchestrator infers the domain and action of each prompt and picks the right model automatically based on plain-English descriptions you define.\nFor alias-based, the config looks like this:\nmodel_aliases:\n  fast-model:\n    target: gpt-4o-mini\n  reasoning-model:\n    target: claude-sonnet-4.5\nWe’ll use preference-aligned routing in this walkthrough.\nFirst, we install Plano. Also, we’re using OpenRouter as our model provider, so one key gives us access to every model we need without managing separate API keys:\npip install planoai\nexport OPENROUTER_API_KEY=sk-or-...\nNote: OpenRouter is one option. Plano works with any OpenAI-compatible endpoint, so you can point it at a self-hosted model running on Ollama or vLLM just by swapping the base_url.\nThen we create the config.\nThe routing preferences tell Plano which tasks go where. For complex reasoning and code, we use\nclaude-sonnet-4.5\n, for everything routine\ngpt-4o-mini\n:\n# plano_config.yaml\nversion: v0.4.0\n\nlisteners:\n  - name: egress_traffic\n    type: model\n    address: 0.0.0.0\n    port: 12000\n    timeout: 30s\n\nmodel_providers:\n  - model: anthropic/claude-sonnet-4.5\n    base_url: https://openrouter.ai/api/v1\n    access_key: $OPENROUTER_API_KEY\n\n  - model: openai/gpt-4o-mini\n    base_url: https://openrouter.ai/api/v1\n    access_key: $OPENROUTER_API_KEY\n    default: true\n\nrouting_preferences:\n  - name: code generation\n    description: writing new code, implementing functions, or building features from scratch\n    models:\n      - anthropic/claude-sonnet-4.5\n\n  - name: complex reasoning\n    description: deep analysis, architectural decisions, debugging hard problems, multi-step reasoning\n    models:\n      - anthropic/claude-sonnet-4.5\n\n  - name: general tasks\n    description: answering questions, explaining concepts, quick lookups, casual conversation\n    models:\n      - openai/gpt-4o-mini\ntracing:\n  random_sampling: 100\n  opentracing_grpc_endpoint: http://localhost:4317\nPlano reads every incoming prompt and routes it to the right model based on these preferences.\nPointing Hermes at Plano\nHermes connects to any OpenAI-compatible endpoint, so we point it at Plano instead of a provider directly:\nFirst, we install Hermes:\ncurl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash\nThen we configure it to use Plano as its endpoint:\nhermes model\n# Select \"Custom endpoint (self-hosted / VLLM / etc.)\"\n# Enter URL: http://localhost:12000/v1\n# Skip API key\nFrom here, every prompt Hermes sends goes to Plano, which classifies it and forwards it to the right model through OpenRouter. Hermes just got a completion back.\nWe start Plano first, so the proxy is ready before Hermes begins sending requests:\nplanoai up plano_config.yaml\nhermes\nHermes is now sending every request through Plano instead of directly to a provider.\nRouting in practice\nNow that both Plano and Hermes are up, we send two different kinds of requests.\nFirst, a simple question:\n> what does the __init__ method do in Python?\nPlano classifies this as “general tasks” and routes it to\ngpt-4o-mini\n. Fast, cheap, more than capable enough for a conceptual question.\nThen something that actually needs horsepower:\n> implement a rate limiter middleware for a FastAPI app using Redis, with per-user and per-endpoint limits\nPlano classifies this as “code generation” and routes it to\nclaude-sonnet-4.5\n.\nThe Hermes session is hitting the same endpoint, but it’s picking a different model automatically based on what the prompt actually needed.\nWe wrote zero branching logic to make that happen. The routing decision lived in the YAML, not in the agent.\nThat covers routing by intent. The next case is when two models can both handle a task equally well, and the only difference is price.\nRouting to the cheapest model that qualifies\nWhen two models can both handle a task equally well, we can let Plano pick based on live pricing instead of the order we wrote them in.\nrouting_preferences:\n  - name: code review\n    description: reviewing, analyzing, and suggesting improvements to existing code\n    models:\n      - anthropic/claude-sonnet-4.5\n      - openai/gpt-4o\n    selection_policy:\n      prefer: cheapest\nPlano reads a pricing catalog before routing and reorders the candidate list by total token cost.\nThe cheaper model answers, but the expensive one stays as a fallback if the cheaper one fails.\nPlano also ships an observability console,\nplanoai obs\n, which shows a per-request USD cost column so we can see exactly which model answered each request and what it cost.\nModel agility, without the rewrite\nArmstrong's team built their routing layer from scratch, inside custom harnesses with engineers assigned to own it.\nMost teams are making the same decisions without any of that infrastructure\nPlano gives you three things out of the box without writing a single line of routing logic.\nPreference-aligned routing automatically sent complex tasks to\nclaude-sonnet-4.5\nand routine ones to\ngpt-4o-mini\n.\nCost-aware selection reached for the cheaper option when two models could both handle the job.\nModel affinity kept the same model pinned across every turn of a multi-step task, so nothing broke mid-session.\nAll of it is configured in YAML instead of inside your coding agent.\nWhen a cheaper model drops for routine tasks, you update one line. When a better one beats Claude on code, you point the preference at it. The agent never changes.\nThe routing decision just moves.\nPlano’s GitHub repo →\n(Don’t forget to star it ⭐️)\nGood day!\nPublished on Jul 6, 2026\nComments\nShare\nCopy link\nShare to X\nShare to Facebook\nShare to Linkedin\nCopied\nPrevious\nRethinking KV Caching For Production Inference\nNext\nAgentic RL: Environments, Trajectories, and the Training",
  "metadata": {
    "title": "How to Reduce LLM Costs by 50-60% Using Model Routing",
    "description": "...explained with code.",
    "keywords": null,
    "author": null,
    "og:site_name": "Daily Dose of Data Science",
    "og:type": "article",
    "og:title": "How to Reduce LLM Costs by 50-60% Using Model Routing",
    "og:description": "...explained with code.",
    "og:url": "https://www.dailydoseofds.com/p/how-to-reduce-llm-costs-by-50-60-using-model-routing/",
    "og:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/07/plano-llm-routing-works.gif",
    "og:image:width": "1200",
    "og:image:height": "900",
    "twitter:card": "summary_large_image",
    "twitter:title": "How to Reduce LLM Costs by 50-60% Using Model Routing",
    "twitter:description": "...explained with code.",
    "twitter:url": "https://www.dailydoseofds.com/p/how-to-reduce-llm-costs-by-50-60-using-model-routing/",
    "twitter:image": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/2026/07/plano-llm-routing-works.gif",
    "twitter:label1": "Written by",
    "twitter:data1": "Avi Chawla",
    "twitter:label2": "Filed under",
    "twitter:data2": "Agents",
    "twitter:site": "@_avichawla",
    "article:published_time": "2026-07-06T15:29:00.000Z",
    "article:modified_time": "2026-07-12T15:30:31.000Z",
    "article:tag": "Agents"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/3b30edc9dac0/code/body-000.txt",
      "chars": 279,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-001.txt",
      "chars": 100,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-002.txt",
      "chars": 101,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-003.txt",
      "chars": 55,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-004.txt",
      "chars": 1016,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-005.txt",
      "chars": 66,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-006.txt",
      "chars": 121,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-007.txt",
      "chars": 35,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-008.txt",
      "chars": 45,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-009.txt",
      "chars": 106,
      "language": "text"
    },
    {
      "file": "attachments/3b30edc9dac0/code/body-010.txt",
      "chars": 241,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "bytes": 11344,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/000.jpg",
      "src": "https://storage.ghost.io/c/3f/df/3fdf6ed2-17ac-4b12-a693-8078bd13e748/content/images/size/w400/2026/03/avi-google.jpg",
      "alt": "Avi Chawla",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!nwx-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46fcbafc-7513-4f03-b287-09ae42499ae2_1500x876.png",
      "bytes": 88456,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/001.png",
      "src": "https://substackcdn.com/image/fetch/$s_!nwx-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F46fcbafc-7513-4f03-b287-09ae42499ae2_1500x876.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!SuHf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F724d7480-3bd6-4487-974d-8971a0e0a590_1492x924.png",
      "bytes": 262023,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/011.png",
      "src": "https://substackcdn.com/image/fetch/$s_!SuHf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F724d7480-3bd6-4487-974d-8971a0e0a590_1492x924.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!HaeX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19891d2-4717-4ccd-9d18-7759e7e9fa80_1508x886.png",
      "bytes": 86003,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/003.png",
      "src": "https://substackcdn.com/image/fetch/$s_!HaeX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19891d2-4717-4ccd-9d18-7759e7e9fa80_1508x886.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!mUN4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbace4ba1-29ae-4581-bc15-4c9f52c9d327_1170x925.png",
      "bytes": 119612,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/004.png",
      "src": "https://substackcdn.com/image/fetch/$s_!mUN4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbace4ba1-29ae-4581-bc15-4c9f52c9d327_1170x925.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!8QSW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5245ee86-5592-4e61-8202-049e9886a4f0_2074x1642.png",
      "bytes": 180823,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/005.png",
      "src": "https://substackcdn.com/image/fetch/$s_!8QSW!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5245ee86-5592-4e61-8202-049e9886a4f0_2074x1642.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!QEJZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe62136c0-c620-4ae3-af84-6a92c6534fc8_1452x1172.png",
      "bytes": 254547,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/006.png",
      "src": "https://substackcdn.com/image/fetch/$s_!QEJZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe62136c0-c620-4ae3-af84-6a92c6534fc8_1452x1172.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!nkid!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff602857e-8978-45e5-9ceb-a73fa56931fb_1006x600.png",
      "bytes": 41832,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/007.png",
      "src": "https://substackcdn.com/image/fetch/$s_!nkid!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff602857e-8978-45e5-9ceb-a73fa56931fb_1006x600.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Pcyz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0628b8cc-31e7-4448-8a20-ee7fb02bd2cc_1456x1116.png",
      "bytes": 266118,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/008.png",
      "src": "https://substackcdn.com/image/fetch/$s_!Pcyz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0628b8cc-31e7-4448-8a20-ee7fb02bd2cc_1456x1116.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!6Mc7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7568d9f6-0422-4326-9e73-accc21ecae5f_2000x734.png",
      "bytes": 68236,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/009.png",
      "src": "https://substackcdn.com/image/fetch/$s_!6Mc7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7568d9f6-0422-4326-9e73-accc21ecae5f_2000x734.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!f0zH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04232418-8063-498b-9d86-6f2bc9014763_2328x1338.png",
      "bytes": 154555,
      "reused": true,
      "local_path": "attachments/3b30edc9dac0/images/010.png",
      "src": "https://substackcdn.com/image/fetch/$s_!f0zH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04232418-8063-498b-9d86-6f2bc9014763_2328x1338.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/katanemo/plano?ref=dailydoseofds.com",
      "label": "here’s the open-source GitHub repo",
      "type": "a"
    },
    {
      "href": "https://x.com/intent/tweet?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-reduce-llm-costs-by-50-60-using-model-routing%2F&text=How%20to%20Reduce%20LLM%20Costs%20by%2050-60%25%20Using%20Model%20Routing",
      "label": "Share to X",
      "type": "a"
    },
    {
      "href": "https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-reduce-llm-costs-by-50-60-using-model-routing%2F",
      "label": "Share to Facebook",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.dailydoseofds.com%2Fp%2Fhow-to-reduce-llm-costs-by-50-60-using-model-routing%2F",
      "label": "Share to Linkedin",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/p/rethinking-kv-caching-for-production-inference/",
      "label": "Previous Rethinking KV Caching For Production Inference",
      "type": "a"
    },
    {
      "href": "https://www.dailydoseofds.com/rl-course-part-12/",
      "label": "Next Agentic RL: Environments, Trajectories, and the Training",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:56Z",
  "source_url": "https://www.dailydoseofds.com/p/how-to-reduce-llm-costs-by-50-60-using-model-routing/",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 9311,
  "expected_body_images": 11,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/3b30edc9dac0/pages/source.html",
  "local_html": "attachments/3b30edc9dac0/pages/article.html",
  "body_base_path": "attachments/3b30edc9dac0/pages/",
  "body_markdown_path": "attachments/3b30edc9dac0/pages/article.md"
}