{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Panagiotis Gkilis",
  "title_listed": "A Hundred Per Cent Success Rate and One Usable Answer",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "工程实践",
  "is_paid_flag": "否",
  "url": "https://dev.to/bedvibe_studios/a-hundred-per-cent-success-rate-and-one-usable-answer-3aj6",
  "publish_date": "2026-08-25",
  "corpus_stage": "",
  "notes": "",
  "id": "74d35f5ef424",
  "collected_at": "2026-10-02T13:52:21",
  "title": "A Hundred Per Cent Success Rate and One Usable Answer - DEV Community",
  "body_markdown": "Originally published at\nai.bedvibe.studio\n.\nI built a gateway that sits in front of several inference backends — authentication, per-key spend quota, token metering, retries, failover, streaming. The ordinary shape. Then, before trusting any of it, I broke it twice on purpose and measured what the broken version reported about itself.\nBoth breakages produced clean logs. That is the part worth writing down.\nThe first fault: a gateway that cannot say no\nI removed admission control — no bound on how many requests may be in flight at once — and sent thirty-two callers at the gateway simultaneously. Each caller was willing to wait\n700 milliseconds\nand would walk away after that.\nEvery single request returned\nHTTP 200\n. Thirty-two successes out of thirty-two. A dashboard reading that gateway would show a flawless service.\nThirty-one of those answers arrived after the caller had stopped waiting.\nOne was usable.\nThe queue had not disappeared when I removed the bound. It had moved. Off the gateway, where I could see it and refuse it, and into the backend, where it was invisible and where everybody's clock was already running. The gateway kept faithfully reporting the outcome of work that no longer had an audience.\nWith a bound and a fast\n429\ninstead: eight answered, twenty-four refused within milliseconds, and\nsix usable answers instead of one\n.\nUnbounded\nBounded\nHTTP 200 returned\n32\n8\nSuccess rate\n100%\n25%\nAnswers that arrived in time\n1\n6\nSix times the useful output, while the headline metric fell by seventy-five points. If you are optimising the success rate, you will optimise your way back into the first column.\nWhere the bound came from\nI did not pick eight because it felt right. I measured the backend until it stopped getting faster:\nConcurrent requests\nThroughput\n1\n189 tokens/sec\n8\n433 tokens/sec\n32\n474 tokens/sec — 9% more, for 3.3x the latency\nThe knee is at eight. Past it you are not buying throughput, you are buying queue. So the admission limit is eight, and it is a measurement rather than a preference — which also means it has an expiry date: change the model or the hardware and the number has to be taken again.\nThe second fault: failover that works in the log and fails for the caller\nEverybody writes failover for a backend that dies. Dying is easy — the connection refuses, you try the next one. So I made a backend go\nslow\ninstead: three seconds to respond, against a caller willing to wait one and a half.\nThe gateway logged\neight successes\n. Every one of them landed at three seconds, after the caller was gone.\nThe healthy backend was never called once.\nThe cause was that my retry budget was a count of attempts plus a per-backend timeout. Neither of those knows whether anybody is still waiting. A retry policy expressed in attempts is a policy about the server's patience, and the server is not the one who is waiting.\nSo the deadline now belongs to the request, and travels with it. While another backend is still available, the current one may spend at most sixty per cent of whatever time remains. You have to\nreserve\ntime for the thing you are failing over to, or the failover is decoration that runs after the deadline it was supposed to protect.\nAttempt-counted\nDeadline-owned\nLogged successes\n8\n8\nArrived in time\n0\n8\np50 latency\n3001 ms\n1046 ms\nNote the first row. The log is identical in both columns. Every observability surface I had said the failover was working, for as long as I only asked it whether requests succeeded.\nThree things I did not plan\nThe two faults above were deliberate. These three found me.\nThe first working call billed 32 tokens and returned an empty string.\nThe local model is a reasoning model, and the server returns its reasoning in a different field from its answer. Read only the answer field and you show the user nothing while charging them for a full generation. Nothing errors, and the token accounting is correct — that is what makes it nasty.\nA concurrency bug that hid inside a green test suite.\nI created the lock that bounds concurrency in the constructor. Python binds those to whichever event loop touches them first, so a second loop makes it throw — on admission, before any model is called, while the health endpoint still says\nok\n. It surfaced as a\nwarning\non a worker thread and the suite reported\n79 passed\n. Then my fix had the same bug in a new costume: I keyed it by the loop's memory address, and Python reuses the address of a closed loop. Both are fixed, and the suite now turns a swallowed background exception into a failure.\nI guessed the wrong hotspot and the trace corrected me.\nI was certain the overhead was the quota check — three database reads on every single request, right there in the hot path. It was\n0.146 ms\n, about four per cent. Three quarters of the overhead was writing one metering row, because the database was flushing to disk before answering. One setting:\n2.999 ms down to 0.230 ms\n, and the gateway's total overhead from 3.47 ms to 0.60 ms.\nThe number I refused to publish\nMeasured end to end, through the whole stack, the gateway added\n+1.26 ms against a noise floor of ±5.38 ms\n.\nThat is not a measurement. It is a number smaller than the uncertainty around it, and quoting it would be quoting noise with a decimal point on it. The benchmark now says so out loud instead of printing the figure.\nIt is also the entire argument for instrumenting inside the system. You cannot find a one-millisecond effect by subtracting two noisy end-to-end numbers. You find it with a span around the thing you suspect — which is how the metering flush turned up, and how I learned my confident guess about the quota check was wrong.\nThe one I was asked to build and didn't\nThe brief said to put vLLM on this machine. I looked at the machine first.\nvLLM has no Windows build, so reaching it means a WSL2 install of roughly ten gigabytes of CUDA — on a box where system RAM, not VRAM, is the bottleneck. And the GPU is not idle: it is shared with a text-to-speech service that sits behind a live API taking real traffic, holding 7.4 of its 16 GB. vLLM's default configuration reserves ninety per cent of the card. The good outcome is that it refuses to start. The bad outcome is that I take down something that earns money, to demonstrate a tool.\nSo I said so, in an architecture decision record, along with what the decision costs:\ncontinuous batching is what vLLM is actually for, and I did not measure it.\nMeasuring something else and calling it vLLM would be a lie with benchmarks attached.\nWhat I built instead is the layer above the engine, and that is the part that transfers. Routing, quota, metering, admission, deadlines and failover know nothing about which engine is underneath. The proof is structural rather than rhetorical: the two backends in it speak completely different protocols and each one is a single file.\nWhat I actually take from this\nSuccess rate is the metric that hides an outage.\nBoth faults produced a clean log. One produced a perfect log. The number I care about now is how many answers arrived while somebody was still there to read them, and every dashboard I build gets that number next to the success rate, not instead of it.\nThe second thing is smaller and I keep relearning it: a fault that returns an error is a fault you will fix this afternoon. A fault that returns a plausible success is one you ship. Both of the faults here were of the second kind, and I only found them because I planted them and then went looking for the specific evidence that they had happened — not because anything alerted.\nNinety-four tests, seven seconds, no model, no GPU, no API key and no network. Zero API calls were spent on any of it.\nSource:\ngithub.com/Mormolykos/basalt\n.\nIf you have a load-shedding or failover story where the log looked fine, I would like to hear it — particularly the ones where the metric that lied was one you had deliberately chosen.\nThe retrieval agent this gateway was built to sit in front of is live, with its architecture and evaluation gate written up at\ntts.bedvibe.studio/portfolio/agent-architecture\n.",
  "metadata": {
    "title": "A Hundred Per Cent Success Rate and One Usable Answer - DEV Community",
    "description": "I removed the backpressure from my own model gateway. Thirty-two callers, thirty-two HTTP 200s, and thirty-one arrived after the caller had given up. Tagged with python, architecture, devops, ai.",
    "keywords": "python, architecture, devops, ai, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/bedvibe_studios/a-hundred-per-cent-success-rate-and-one-usable-answer-3aj6",
    "og:title": "A Hundred Per Cent Success Rate and One Usable Answer",
    "og:description": "I removed the backpressure from my own model gateway. Thirty-two callers, thirty-two HTTP 200s, and thirty-one arrived after the caller had given up.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffnqe26278tvhl5vqnfc9.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "A Hundred Per Cent Success Rate and One Usable Answer",
    "twitter:description": "I removed the backpressure from my own model gateway. Thirty-two callers, thirty-two HTTP 200s, and thirty-one arrived after the caller had given up.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ffnqe26278tvhl5vqnfc9.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://ai.bedvibe.studio/success-rate/",
      "label": "ai.bedvibe.studio",
      "type": "a"
    },
    {
      "href": "https://github.com/Mormolykos/basalt",
      "label": "github.com/Mormolykos/basalt",
      "type": "a"
    },
    {
      "href": "https://tts.bedvibe.studio/portfolio/agent-architecture/",
      "label": "tts.bedvibe.studio/portfolio/agent-architecture",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:45Z",
  "source_url": "https://dev.to/bedvibe_studios/a-hundred-per-cent-success-rate-and-one-usable-answer-3aj6",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 8084,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/74d35f5ef424/pages/source.html",
  "local_html": "attachments/74d35f5ef424/pages/article.html",
  "body_base_path": "attachments/74d35f5ef424/pages/",
  "body_markdown_path": "attachments/74d35f5ef424/pages/article.md"
}