{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Arvind Narayanan & Sayash Kapoor / AI Snake Oil",
  "title_listed": "Towards a Science of AI Agent Reliability",
  "keywords": "Agent可靠性,能力-可靠性差距,评估基准",
  "domain": "机器学习",
  "content_type": "研究分析",
  "is_paid_flag": "否",
  "url": "https://www.normaltech.ai/p/new-paper-towards-a-science-of-ai?utm_source=publication-search",
  "publish_date": "2025-02",
  "corpus_stage": "RLHF/SFT",
  "notes": "AI Agent可靠性科学化评估，量化能力-可靠性差距",
  "id": "83b88495c4de",
  "collected_at": "2026-10-01T19:51:22",
  "title": "New Paper: Towards a science of AI agent reliability",
  "body_markdown": "By Stephan Rabanser, Sayash Kapoor, Arvind Narayanan\nSuppose you hear about a new AI agent for improving productivity — by making purchases, or writing code, or sending emails, or handling a customer on your behalf. Should you trust it? Can the agent do the job reliably enough? After all, there are\nmany\nhorror\nstories\nof agents going wrong.\nSurprisingly, even though the lack of reliability of AI agents is well known, right now the AI industry doesn’t have good tools for measuring reliability, or even a good definition of reliability.\nArvind and Sayash have long been\nthinking\nabout this. Last fall, we were joined by postdoctoral researcher\nStephan Rabanser\n, whose PhD looked at the reliability question in simpler, more traditional AI systems. We recruited a few other independent researchers, and have released what we hope is a comprehensive measurement of reliability. Our draft paper is called\nTowards a Science of AI Agent Reliability\n.\nWe borrowed insights from many other fields, such as nuclear and aviation safety. We were able to decompose reliability into 12 different dimensions. Evaluating 14 models on two complementary benchmarks, we found that nearly two years of rapid capability progress have produced only modest reliability gains. See our interactive dashboard\nhere\n.\nWhile our findings are tentative at this stage, we hope they can help explain the\npuzzlement\namong many in the industry as to why the economic impacts of AI agents have been gradual, even though they are crushing capability benchmarks.\n1\nTo help the community track reliability systematically, we plan to launch an AI agent “reliability index”. We hope this will stimulate researchers and industry to invest effort into improving reliability.\nTable of Contents\nAccuracy isn’t enough: four dimensions of reliability\nCapability gains are rapid, but improvements in reliability are modest\nWhy we could be wrong\nWhat should deployers do differently?\nWhat should researchers and developers do differently?\nWhat do our findings mean for AI progress?\nFurther reading\nAccuracy isn’t enough: four dimensions of reliability\nWhen we consider a coworker to be reliable, we don’t just mean that they get things right most of the time. We mean something richer:\nThey get it right consistently, not right today and wrong tomorrow on the same thing (Consistency)\nThey don’t fall apart when conditions aren’t perfect (Robustness)\nThey tell you when they’re unsure rather than confidently guessing (Calibration)\nWhen they do mess up, their mistakes are more likely to be fixable than catastrophic (Safety)\nUnfortunately, AI agents are evaluated based on a single number, the average success rate at the task. That number has been going up quickly on many tasks over the last two years, which is why there’s so much excitement about deploying agents.\nSafety-critical engineering fields (aviation, nuclear, automotive) figured out decades ago that reliability is not the same as average performance. These fields independently converged on the above four dimensions: consistency, robustness, predictability, and safety (the frequency and severity of failures).\nFor example, nuclear reactor protection systems\nmust respond identically\nevery time conditions warrant shutdown. Automotive safety testing evaluates\nresponses to sensor failures\nand adverse weather. Nuclear risk assessment models thousands of failure modes and\nquantifies their probabilities\n. Aviation\ntargets one catastrophic error per billion flight hours\n.\nCapability gains are rapid, but improvements in reliability are modest\nWe refined and decomposed these four high-level dimensions into twelve metrics. We then tested agents based on 14 models from OpenAI, Google, and Anthropic, spanning 18 months of releases. We looked at two complementary benchmarks: a general assistant benchmark (GAIA) and a customer service simulation benchmark (TauBench). We ran each task five times, with instructions paraphrased. We injected faults in the tools and environment to measure robustness to such failures, and elicited the agents’ confidence at having solved the task to measure calibration. In total, we executed 500 overall benchmark runs.\nWe found that reliability has improved only modestly over 18 months, while accuracy improved substantially. All three major providers cluster together, so this appears to be an industry-wide limitation (though there are some cases where Anthropic’s models outperform OpenAI’s and Google’s).\nMore specifically, we measured the following criteria:\nConsistency\n: Agents that can solve a task often fail on repeated attempts under identical conditions. Many models have trouble giving a consistent answer, with outcome consistency scores ranging from 30% to 75% across the board.\nRobustness\n: Most models handle genuine technical failures (server crashes, API timeouts) gracefully. But if we rephrase the instructions with the same semantic meaning, performance drops substantially.\nPredictability\n: Agents are not good at knowing when they’re wrong. This is the weakest dimension across the board. When agents report confidence, it often carries little signal. On one benchmark, most models couldn’t distinguish their correct predictions from incorrect ones better than chance.\nSafety\n: Recent models are noticeably better at avoiding constraint violations, though financial errors, such as incorrect charges, remain a common failure mode. We use safety narrowly to mean bounded harm when failures occur, not broader concerns like alignment. We are still iterating on how we measure safety, so we report it separately from the aggregate reliability score.\nImpact of scaling\n: Bigger models aren’t uniformly more reliable. Scaling up improves some dimensions (calibration, robustness) but can hurt consistency. Larger models with richer behavioral repertoires sometimes show more run-to-run variability.\nWhy we could be wrong\nOur view is that reliability lags capability, and that reliability will remain a barrier to deployment unless researchers and developers focus effort on improving reliability as a separate dimension from accuracy. There are three reasons why we could be wrong.\nFirst, there is some subjectivity in our dimensions and metrics. We have tried to minimize this by grounding our analysis in existing engineering fields. And we finalized our list of metrics\nbefore\nperforming experiments, in order to prevent our hypothesis about slow reliability progress from influencing our selection of metrics. Still, we welcome other researchers to suggest alternative ways to measure reliability, and emphasize that\nour findings are tentative at this stage.\nSecond, maybe reliability won’t matter if accuracy gets high enough. Our metrics are carefully crafted so that accuracy gains don’t automatically lead to reliability gains. Broadly speaking, accuracy is about the rate of failures while reliability is about the nature of failures. But if an agent is accurate 99% of the time, maybe we can tolerate 1% error even if it is completely unpredictable. We disagree. Our view is that for autonomous operation in high-stakes contexts, we need 3-5 “nines” of performance — 99.9% to 99.999% accuracy — in order for reliability to become a non-issue, and we don’t think LLM-based agents are on track to reach such a threshold. But time will tell.\nThird, reliability progress being slower than accuracy doesn’t necessarily mean that it is slow in absolute terms. If we project the current linear trend forward, agents will reach 100% reliability in just three years! We don’t think a linear model makes sense, in part because we expect each order of magnitude decrease in\nunreliability\n(1-reliability) to be as hard as the previous one. That is, we expect the jump from 90 to 99% reliability to be about as hard as the jump from 99 to 99.9% reliability, and so on. But again, we just have to wait and see.\nSuppose we’re right. There are important implications for deployers, researchers & developers, and for those tracking the pace of AI progress. Let’s discuss each in turn.\nWhat should deployers do differently?\nClearly distinguish automation from augmentation\n. A coding assistant that occasionally suggests wrong variable names is annoying; an autonomous agent managing an industrial plant yielding highly variable output is unacceptable. The difference is whether the agent is used to augment a human’s creativity or directly make decisions. Augmentation tools (copilots, search assistants) get a reliability “discount” because someone reviews the output. In some augmentation use-cases, reliability might actually be undesirable. For example, a creative writing assistant that produces the same story every time would be terrible at its job.\nIncidentally, how well agents collaborate with humans is woefully under-theorized and under-measured. There is some early work on\nevaluating\nhuman-LM\ninteraction, but both efforts predate autonomous agents, and we are not aware of any equivalent work studying how agents collaborate with humans over multi-step tasks. Uplift studies offer one useful lens, but a broader agent-focused effort is overdue.\nConsider reliability for making release decisions\n. For automation tools (unattended workflows, customer-facing bots), reliability is non-negotiable. Deployers should consider requiring reliability thresholds before moving from sandbox to production, the way aviation requires certification before service. There are many other practices to borrow from such domains, such as building an\nincident\n-\nreporting\nculture\naround agent failures.\nWhile the metrics we have identified are broadly useful as a starting point for understanding reliability, deployers should build their own internal evaluations tailored to their specific context and datasets.\nWhat should researchers and developers do differently?\nBenchmarks drive progress in AI. A year and a half ago, our paper\nAI Agents that Matter\nshowed there was a big gap between what agent benchmarks measured and what matters in practice. Our new paper shows that the gap persists. To fix this disconnect, both AI evaluation and AI development practices need to change course.\nMeasure reliability\n. The current approach of running a benchmark once and reporting the accuracy number is a shallow, superficial performance measure. It is comparable to stress-testing a car once in perfect weather and declaring it safe if it passes. When evaluating agents, we need to test them for multiple runs (testing for variance in outcomes), under different conditions (evaluating adaptability), and on an ongoing basis (re-test as models and environments change). We call for reporting reliability profiles alongside accuracy, not instead of it.\nUnderstand and improve reliability\n. Our experiments suggest that consistency and predictability are the biggest gaps preventing models from being more reliable. Agent developers should consider improving these weak points explicitly, possibly via targeted optimization or improved scaffolding. In particular, agents should be able to recognize when they are likely to fail and say so, and recover gracefully when they do fail. More speculative ideas include agents that explore different strategies during development but follow a consistent execution plan when deployed, rather than solving the same task differently each time, simultaneously delivering the best of\nagents and workflows\n.\nWhat do our findings mean for AI progress?\nThe capability-reliability gap could be one reason why we are not seeing any of the rapid labor-market effects that Artificial General Intelligence has been predicted to bring about. It is not the only one. A recent UK AISI\nreport\nidentified six barriers to AGI. However, the report discussed each at a high level.\nOur paper can be seen as putting flesh on the bones of one of these barriers — reliability. None of the four dimensions we identify can be considered solved at this point in time, and only two of the individual metrics, shown in green in the figure, can be considered (tentatively) solved.\nFuture work on other barriers to AGI may reveal many other dimensions of performance that must be improved before AI agents can be widely deployed.\nThere is much work to be done in fleshing out the other barriers to AGI, analogous to our analysis of reliability. Our hunch is that this will reveal many other dimensions and metrics on which progress has been slow. The gazillion-dollar question is whether agents will get better across the board through general methods such as inference scaling and reinforcement learning, or whether painstaking work will be required to improve individual dimensions of reliability, adaptability, originality, and so on.\nSubscribe to receive evidence-based analysis of AI developments and the future of AI.\nSubscribe\nFurther reading\nThe full 66-page paper is available on\narXiv\nand includes definitions of the metrics we propose, details on our experimental setup, as well as an extended discussion on our recommendations and proposed future work. The authors are Stephan Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. Alongside the paper, we also provide an\ninteractive dashboard\nof our results to enable practitioners to drill down into individual models, benchmarks, and reliability dimensions. The code to reproduce our experiments is available on\nGitHub\n.\nThis paper is part of our broader effort to advance the\nScience of AI Agent Evaluation\n. Key publications include\nAI Agents that Matter\nand\nHolistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation\n, recently accepted to ICLR 2026.\nOur paper can be seen as building on work that aims to extend the scope of what we measure through benchmarks. For example,\nHELM\nintroduced 7 metrics, including calibration and robustness, and used these to evaluate dozens of language models on prominent benchmarks.\n1\nOf course, this is not the only factor. Benchmarks continue to\nstruggle\nwith data contamination, so they might overestimate capability progress.\nAI as Normal Technology\nprovides a useful framework to understand the barriers between capability progress and economic impacts. Reliability falls into the products/applications stage. Human, social, and organizational factors lie downstream. As a case study of how serious these can be, see our recent essay\nAI Won’t Automatically Make Legal Services Cheaper\n.",
  "metadata": {
    "title": "New Paper: Towards a science of AI agent reliability",
    "description": "Quantifying the capability-reliability gap",
    "keywords": null,
    "author": "Sayash Kapoor",
    "og:type": "article",
    "og:title": "New Paper: Towards a science of AI agent reliability",
    "og:description": "Quantifying the capability-reliability gap",
    "og:image": "https://substackcdn.com/image/fetch/$s_!vhlu!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e9d4b87-9068-4764-b50a-9c458b45252b_1486x960.jpeg",
    "og:url": "https://www.normaltech.ai/p/new-paper-towards-a-science-of-ai",
    "twitter:title": "New Paper: Towards a science of AI agent reliability",
    "twitter:description": "Quantifying the capability-reliability gap",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!zd53!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Faisnakeoil.substack.com%2Fapi%2Fv1%2Fpost_preview%2F189010640%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-02-24T18:30:30.259Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!KXpN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd904192c-a01e-42fb-8990-b46f387f4690_2120x1276.jpeg",
      "bytes": 47010,
      "reused": false,
      "local_path": "attachments/83b88495c4de/images/body-53e1e9e43ad283720518.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!KXpN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd904192c-a01e-42fb-8990-b46f387f4690_2120x1276.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!MID-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f64a3e5-d811-480d-9003-b2fcd0bc6e92_1522x974.png",
      "bytes": 92169,
      "reused": false,
      "local_path": "attachments/83b88495c4de/images/body-06365b98d6068d238a31.png",
      "src": "https://substackcdn.com/image/fetch/$s_!MID-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f64a3e5-d811-480d-9003-b2fcd0bc6e92_1522x974.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://www.businessinsider.com/replit-ceo-apologizes-ai-coding-tool-delete-company-database-2025-7",
      "label": "many",
      "type": "a"
    },
    {
      "href": "https://www.washingtonpost.com/technology/2025/02/07/openai-operator-ai-agent-chatgpt/",
      "label": "horror",
      "type": "a"
    },
    {
      "href": "https://themarkup.org/artificial-intelligence/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law",
      "label": "stories",
      "type": "a"
    },
    {
      "href": "https://sites.google.com/princeton.edu/agents-workshop",
      "label": "thinking",
      "type": "a"
    },
    {
      "href": "https://rabanser.dev/",
      "label": "Stephan Rabanser",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2602.16666",
      "label": "Towards a Science of AI Agent Reliability",
      "type": "a"
    },
    {
      "href": "https://hal.cs.princeton.edu/reliability/",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!KXpN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd904192c-a01e-42fb-8990-b46f387f4690_2120x1276.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.dwarkesh.com/p/thoughts-on-ai-progress-dec-2025",
      "label": "puzzlement",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/i/189010640/accuracy-isnt-enough-four-dimensions-of-reliability",
      "label": "Accuracy isn’t enough: four dimensions of reliability",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/i/189010640/capability-gains-are-rapid-but-improvements-in-reliability-are-modest",
      "label": "Capability gains are rapid, but improvements in reliability are modest",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/i/189010640/why-we-could-be-wrong",
      "label": "Why we could be wrong",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/i/189010640/what-should-deployers-do-differently",
      "label": "What should deployers do differently?",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/i/189010640/what-should-researchers-and-developers-do-differently",
      "label": "What should researchers and developers do differently?",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/i/189010640/what-do-our-findings-mean-for-ai-progress",
      "label": "What do our findings mean for AI progress?",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/i/189010640/further-reading",
      "label": "Further reading",
      "type": "a"
    },
    {
      "href": "https://www.nrc.gov/docs/ML0705/ML070550085.pdf",
      "label": "must respond identically",
      "type": "a"
    },
    {
      "href": "https://www.iso.org/standard/68383.html",
      "label": "responses to sensor failures",
      "type": "a"
    },
    {
      "href": "https://www.nrc.gov/reading-rm/doc-collections/fact-sheets/probabilistic-risk-asses",
      "label": "quantifies their probabilities",
      "type": "a"
    },
    {
      "href": "https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC%2023.1309-1E.pdf",
      "label": "targets one catastrophic error per billion flight hours",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2405.10632v2",
      "label": "evaluating",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2508.14231",
      "label": "incident",
      "type": "a"
    },
    {
      "href": "https://cset.georgetown.edu/publication/ai-incidents-key-components-for-a-mandatory-reporting-regime/",
      "label": "reporting",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2503.16861",
      "label": "culture",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2407.01502",
      "label": "AI Agents that Matter",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/VirtualOasis/agents-vs-workflows-en",
      "label": "agents and workflows",
      "type": "a"
    },
    {
      "href": "https://www.aisi.gov.uk/blog/mapping-the-limitations-of-current-ai-systems",
      "label": "report",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!MID-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f64a3e5-d811-480d-9003-b2fcd0bc6e92_1522x974.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/steverab/hal-harness",
      "label": "GitHub",
      "type": "a"
    },
    {
      "href": "https://sage.cs.princeton.edu/",
      "label": "Science of AI Agent Evaluation",
      "type": "a"
    },
    {
      "href": "https://www.normaltech.ai/p/new-paper-ai-agents-that-matter",
      "label": "AI Agents that Matter",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2510.11977",
      "label": "Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2211.09110",
      "label": "HELM",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2602.12413",
      "label": "struggle",
      "type": "a"
    },
    {
      "href": "https://knightcolumbia.org/content/ai-as-normal-technology",
      "label": "AI as Normal Technology",
      "type": "a"
    },
    {
      "href": "https://www.lawfaremedia.org/article/ai-won-t-automatically-make-legal-services-cheaper",
      "label": "AI Won’t Automatically Make Legal Services Cheaper",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:06Z",
  "source_url": "https://www.normaltech.ai/p/new-paper-towards-a-science-of-ai?utm_source=publication-search",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2602.16666",
      "bytes": 1901122,
      "reused": false,
      "local_path": "papers/repair/580e6dcf40e8cefafe4f.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.nrc.gov/docs/ML0705/ML070550085.pdf",
      "bytes": 134613,
      "reused": false,
      "local_path": "papers/repair/965e2cab921d26d4ee8f.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC%2023.1309-1E.pdf",
      "bytes": 585828,
      "reused": false,
      "local_path": "papers/repair/16ec4d2b6031704d8dc9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2405.10632v2",
      "bytes": 387799,
      "reused": false,
      "local_path": "papers/repair/186d240a752380f3d290.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2508.14231",
      "bytes": 405484,
      "reused": false,
      "local_path": "papers/repair/f11679041bda0883ea0a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2503.16861",
      "bytes": 6163511,
      "reused": false,
      "local_path": "papers/repair/6b276965dcbe10103aaf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2407.01502",
      "bytes": 1235369,
      "reused": false,
      "local_path": "papers/repair/952f6b5abbf2d346beb4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2510.11977",
      "bytes": 12428285,
      "reused": false,
      "local_path": "papers/repair/d06e2006c7d212508fa6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2211.09110",
      "bytes": 6045688,
      "reused": false,
      "local_path": "papers/repair/1daba803f8c6184b06f8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2602.12413",
      "bytes": 2998326,
      "reused": false,
      "local_path": "papers/repair/58792e32076f0b488ad0.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 14422,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/83b88495c4de/pages/source.html",
  "local_html": "attachments/83b88495c4de/pages/article.html",
  "body_base_path": "attachments/83b88495c4de/pages/",
  "body_markdown_path": "attachments/83b88495c4de/pages/article.md"
}