{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Debashish Ghosal",
  "title_listed": "A Certification That Changes Every Run Is a Coin Flip With a Signature",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/debashish_ghosal/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature-bj9",
  "publish_date": "2026-09-27",
  "corpus_stage": "",
  "notes": "",
  "id": "e59d41af85c9",
  "collected_at": "2026-10-02T14:05:50",
  "title": "A Certification That Changes Every Run Is a Coin Flip With a Signature - DEV Community",
  "body_markdown": "The plan was clean, and I believed every word of it: three real agents I had already built — a repo guardian, a release-notes drafter, an incident commander — would register in\nHivePlane\n, the control plane I'd just spent six weeks building, certify against their corpora, and prove the certified loop on\nreal\nworkloads.\nDay one, all three failed. None of the failures were the control plane's fault — which is exactly what made them interesting.\nIf you have ever tried to run an agent outside the repo it was born in, you already know where this is going.\nThe three failures\nAgent\nWhat I expected\nWhat actually happened\nrelease-narrator\nImport and certify\nfrom langgraph.checkpoint.sqlite import SqliteSaver\n— the module isn't installed in the control-plane environment\nai-incident-commander\nImport and certify\nExpects an installed\nincident_commander\npackage; imports\nincident_commander.ingest.input_dir\n, which doesn't exist in the downloaded tree\nall three\nGoverned execution\nBoth import the external\nopenai\nSDK — absent by design; HivePlane has its own provider seam, and agents cannot call providers directly\nThese were\nmy own agents\n, from my own repos, and they weren't drop-in runnable. That's a real finding about agent portability in general — \"works in my repo\" is not an interface — but it wasn't the finding the field test existed to produce.\nThe worse problem: nondeterminism\nThe trio that\ndid\nrun made certification flaky. All three drive human-in-the-loop flows whose outputs depend on the LLM, and the early field-test runs recorded the failure mode plainly:\nthe real local model misclassified a bug-fix task as\nchangelog\nand failed certification nondeterministically across runs.\nRead that again from the platform's point of view. A certification result that changes run to run isn't a certification. It's a coin flip with a signature on it. My thresholds were deterministic; my signal wasn't.\nThe decision: what is the system under test?\nI had to answer one question honestly:\nis this field test supposed to measure the agents, or the control plane?\nThe thesis of HivePlane is the loop — register, certify, gate admission, enforce budget and policy, pause and resume, deliver, audit. The agent is the workload; the plane is the product. So the Tier 1 subjects became\ndeterministic real agents\nwired through thin shims:\nTier 1 subject\nAdapter\nWhat it exercises\nsupport-agent\n(exectrace\nagent-raw\n)\nraw-worker\nRead-first tool calls through the boundary, escalation to a destructive tool, 40 KB output truncation\neval-judge\n(exectrace judge graph)\nlanggraph\nStateGraph, cycles, human-review\ninterrupt()\n, durable checkpointer resume\nBoth are real agents — real code, real adapter dispatch, real policy and tool boundary — with a mock KB, mock tools, and a mock judge. Determinism is a\nfeature of the test design\n: the same seed always produces the same category of run.\ntip:\nThe model is still on the path where it matters. Identity binding, the provider seam, and the model-swap gate all run during certification and runs. What is mocked is the\njudge's opinion\n, not the plumbing.\nThe result: certification became repeatable\nFrom the\nfield test report\n:\nS1 passed identically in three consecutive stack runs\n— same tasks, same verdicts, same pass rates. Production certifications: support-agent 6/6 at p95 67 ms, eval-judge 4/4 at p95 126 ms, both at the 0.90 threshold with signed Ed25519 attestations.\nThat is what a benchmark gate needs to be before it can gate anything:\nboring\n. The signal now measures the control plane, not model drift.\nWhat the field test caught that unit tests couldn't\nTwo findings only a live stack produces:\nThe egress allowlist bit.\nThe escalation task issues a destructive\npagerduty.acknowledge\ncall — and\napi.pagerduty.com\nwasn't in the manifest's\nsandbox.egress.allow\n. The call was denied and the run died with the symptom\nexpected status='escalated', got None\n. Unit tests mocked the host; only the live stack proved the allowlist was incomplete. Every host an agent's tools target must be allowed.\nUnder-instrumentation reads as a hang.\nTwo scenarios — destructive approval, and pause → restart → resume — were repeatedly aborted mid-run because they wrote no evidence until after the final poll, so any abort erased the diagnosis. The fix was step-wise evidence:\nsubmitted.json\n,\npaused.json\n,\napprovals.json\n,\nresumed.json\nwritten\nbefore each boundary\n. One subtlety worth remembering:\nPOST /runs/{id}/resume\ncan return 409 when the approval's automatic re-dispatch already advanced the run — success is defined by the run reaching\ncompleted\n, which it did. The lesson, not the scenario, was the problem.\nThe restart scenario itself is worth naming: a paused LangGraph run survived a full\ndocker compose restart api\n— event log intact, 8 events, re-attached from a durable checkpoint on the volume — then resumed to\ncompleted\n. The hard part worked on the first attempt, every time. It only ever\nlooked\nstuck.\nWhat I learned\n\"Drop-in runnable\" is an interface you have to design.\nMy own three agents failed in a new environment for three different reasons — missing dependency, missing package layout, wrong SDK. If your agent can't run outside its birth repo, it can't be operated by a fleet. That's a workload problem, not a platform problem.\nDeterminism is a certification requirement, not a convenience.\nA benchmark that returns different verdicts for the same artifact can't be a gate — you can't build \"requires 'certified'\" on top of noise. Mock the judge, never the plumbing.\nThe field test's job is to catch what unit tests structurally cannot.\nEgress allowlists, restart durability, approval flows — these only exist live. If your field test only re-proves your unit suite, you built an expensive unit test.\nWrite evidence before the boundary, not after the verdict.\nEvery scenario that \"hung\" was actually a scenario whose evidence hadn't been written yet. The same applies to any long-running verification job you own.\nWhat it doesn't prove\nThe heavyweight trio stays on disk as references — documented, not deleted, with their import incompatibilities recorded in\nresults/NOTES.md\n.\nDeterministic subjects mean the certification signal is\nstrong but narrow\n— it proves the loop, not your model. The cloud-profile run, with real prices and real drift, is the next release's work.\nThe S6/S8 verdicts were recorded in standalone runs against the live stack, consolidated with the sweep; a single uninterrupted\nscripts/field-test.sh\nregenerates everything from one run.\nReferences\nField test report (v0.1.0)\n— the rewire, the three identical certification runs, the per-scenario evidence\nDocker test report\n— the 25/25 container layer, including the restart-durability test\nresults/NOTES.md\n— raw run history, aborts included\nNext in the series: the four deliberately bad agents I aimed at my own admission gate, and the exact refusal each one earned.\nWhat's the most expensive \"it works on my machine\" failure you've hit with an agent — a missing dependency, a package layout, or something worse?",
  "metadata": {
    "title": "A Certification That Changes Every Run Is a Coin Flip With a Signature - DEV Community",
    "description": "The plan was clean, and I believed every word of it: three real agents I had already built — a repo... Tagged with ai, testing, llm, certification.",
    "keywords": "ai, testing, llm, certification, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/debashish_ghosal/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature-bj9",
    "og:title": "A Certification That Changes Every Run Is a Coin Flip With a Signature",
    "og:description": "The plan was clean, and I believed every word of it: three real agents I had already built — a repo...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F54agy06ui9477ykjuqjj.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "A Certification That Changes Every Run Is a Coin Flip With a Signature",
    "twitter:description": "The plan was clean, and I believed every word of it: three real agents I had already built — a repo...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F54agy06ui9477ykjuqjj.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/deghosal-2026/hiveplane",
      "label": "HivePlane",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/hiveplane/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT.md",
      "label": "field test report",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/hiveplane/blob/main/field_test/v0.1.0/results/NOTES.md",
      "label": "results/NOTES.md",
      "type": "a"
    },
    {
      "href": "https://github.com/deghosal-2026/hiveplane/blob/main/docs/field-test/v0.1.0/DOCKER_TEST_REPORT.md",
      "label": "Docker test report",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:50:09Z",
  "source_url": "https://dev.to/debashish_ghosal/a-certification-that-changes-every-run-is-a-coin-flip-with-a-signature-bj9",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7061,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/e59d41af85c9/pages/source.html",
  "local_html": "attachments/e59d41af85c9/pages/article.html",
  "body_base_path": "attachments/e59d41af85c9/pages/",
  "body_markdown_path": "attachments/e59d41af85c9/pages/article.md"
}