{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Edward Izgorodin",
  "title_listed": "Interpretability Built an Instrument to Prove a Signal Is Actually Used. Agent Memory Has Nothing Like It.",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/izgorodin/interpretability-built-an-instrument-to-prove-a-signal-is-actually-used-agent-memory-has-nothing-2mnf",
  "publish_date": "2026-09-05",
  "corpus_stage": "",
  "notes": "",
  "id": "ff05b9919043",
  "collected_at": "2026-10-02T14:03:20",
  "title": "Interpretability Built an Instrument to Prove a Signal Is Actually Used. Agent Memory Has Nothing Like It. - DEV Community",
  "body_markdown": "A model can contain a decodable signal and not use it to produce its answer.\nThat sentence is the whole problem with interpretability, and it took me an embarrassingly long time to feel its weight. A linear probe recovers a concept from an activation. A sparse autoencoder isolates a direction that correlates with it. Neither shows that the direction did any work. The information is present. Whether it is load-bearing is a different question, and the two get conflated constantly.\nAnthropic's Global Workspace paper is built around an instrument designed to separate them, and the instrument is more interesting to me than the findings. It is called the Jacobian lens.\nWhat it replaces\nThe logit lens decodes a hidden state by pushing it through the model's unembedding matrix. Cheap, unsupervised, and brittle. In the workspace paper's framing, that amounts to setting the Jacobian to the identity: you assume the path from this layer to the output is a straight line, when it is not.\nThe tuned lens improves on that by training a small affine probe per block to match the output distribution. More reliable, and still correlational: it tells you what a layer's state resembles, not what the model does with it.\nThe Jacobian lens replaces the identity with the model's own averaged Jacobian from each layer to the final layer, computed over many prompts. That change is what makes its directions testable rather than merely readable.\nFour operations, and only one of them matters for trust\nREAD\nprojects a hidden state through the Jacobian, normalizes, and unembeds it into a vocabulary distribution.\nWRITE\nadds a scaled lens vector into the residual stream.\nPATCH\nswaps the coordinates of two lens vectors inside the activation.\nABLATE\nzeroes the projection of the residual stream onto the most active lens vectors across a chosen band of layers.\nREAD is the one that looks like the other lenses. The other three are the point. An interpretability claim earns trust when someone\nedits the internal quantity and observes the predicted behavior change, selectively\n.\nThe selectivity is not decoration. Swapping the internal pattern for \"spider\" to \"ant\" changes the model's arithmetic answer from eight to six. Swapping Soccer to Rugby flips the verbal report. A rhyming-couplet plan swap from Fight to Light shifts words that appear\nearlier\nin the line. In each case adjacent capabilities stay intact, which is what separates a real intervention from breaking the model and calling the damage a finding.\nTwo numbers that travel badly\nThis is where I want to slow down, because both of these get repeated without their scope, and I have seen both quoted flat.\nThe hundredfold connectivity.\nThe sentence people cite is real: J-space patterns are read from and written to far more than ordinary patterns, \"in some parts of the network by a factor of about a hundred.\" That phrasing is from Anthropic's plain-language research page. The technical paper reports the same property\nqualitatively\n, saying these vectors compose with downstream weights more broadly than other representational vectors. If you are citing the number, cite the overview, and keep \"in some parts of the network\" attached to it.\nThe ablation collapse.\nRemove J-space and multi-step reasoning drops to near zero. That is the headline, and it is accurate for that task. What travels less well is the other half of the same result: without its J-space the model still \"speaks fluently, classifies sentiment, answers multiple-choice questions, and pulls facts out of passages roughly as before.\"\nSo the collapse is selective, not general. And there is a further wrinkle worth keeping: on grade-school math, chain-of-thought solving is substantially more robust to ablation than answering the same problems directly. The model externalizes its reasoning onto the page, and what is on the page does not need the workspace.\nThat last detail is the most interesting sentence in the whole area, and it almost never gets quoted.\nThe authors' limits, in their words\nThe paper is unusually forthright, and the limits are sharper than most summaries admit.\nThe lens only identifies vectors for concepts that are\nsingle tokens\nin the vocabulary. \"San Francisco\" and \"machine learning\" have no single unembedding direction. The limit is structural: the vocabulary indexes the lens.\nSmall integers fail to swap reliably, and the cause is unresolved.\nThe paper leaves both hypotheses open: the model may compute over small integers outside the workspace, or its working representation of them may simply not align with the lens vectors. The evidence does not choose.\nIn roughly the\nfirst third of the model\n, readouts are noisy and largely uninterpretable. Whether that silence is genuine absence or lens degeneracy is explicitly left open.\nAnd the summary judgment is theirs, not mine: the Jacobian lens is \"an imperfect tool, which we believe only approximately and incompletely captures the model's underlying workspace structure.\"\nThere is also a circularity worth naming out loud. The lens is constructed to find verbalizable content, and then evaluated for further properties of that content.\nWhy any of this matters outside a weights lab\nHere is the part I keep turning over.\nAn agent memory system has exactly the same problem, one layer up. You retrieve a fact and put it in the context. Did the model use it, or did it answer from its prior and leave your retrieved atom sitting there inert?\nRetrieval metrics do not answer that. Hit rate says the record could be reached. Relevance scores say it resembled the query. Neither is the question. The question is whether the atom was load-bearing, and it is the same distinction the Jacobian lens was built to settle inside the network.\nThe instrument does not transfer.\nIt needs weights, activations, and gradients. A production assistant is a black box: no weights, no activations, often not even per-token logprobs.\nThe obvious analogue is behavioral rather than mechanistic. Run the reader three times with the atom\npresent\n,\nremoved\n, and\nminimally corrupted\n, and judge the answer each time. If the atom entered the working set, the full condition differs from the other two in a predictable way. It is causal mediation applied to retrieval, and it would yield per-atom labels rather than a score: load-bearing, redundant, inert, harmful, contested.\nI want to be careful about the status of that idea.\nIt is a research question, not a method, and nobody ships it.\nThe framing is mine and it is not a product.\nThe caveats are real, too. The removed arm is ambiguous: an atom can look ignored because it was unused, or because the answer was overdetermined by the model's prior and the atom simply had nothing left to contribute. The only mitigation I can think of is probing with facts the base model cannot know, which is more work than it sounds and which nobody has quantified.\nSo the honest summary is not a proposal. It is a gap.\nInterpretability spent years learning that a decodable signal is not a used signal, and built an instrument that can tell the difference by intervention. Agent memory has not had that argument yet. We measure whether a fact can be found and call it done, and the distance between storing a fact and proving it was used is currently unmeasured.\nDisclosure: I work on\nMnemoverse\n, a memory engine for AI agents, so the last section is the one to weigh most carefully. Every claim about the Jacobian lens above comes from the paper or Anthropic's own research page, both linked, and the longer technical explainer is on the\noriginal page\n. Sources:\nthe workspace paper\n,\nAnthropic's overview\n.",
  "metadata": {
    "title": "Interpretability Built an Instrument to Prove a Signal Is Actually Used. Agent Memory Has Nothing Like It. - DEV Community",
    "description": "A model can contain a decodable signal and not use it to produce its answer.  That sentence is the... Tagged with ai, machinelearning, llm, agents.",
    "keywords": "ai, machinelearning, llm, agents, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/izgorodin/interpretability-built-an-instrument-to-prove-a-signal-is-actually-used-agent-memory-has-nothing-2mnf",
    "og:title": "Interpretability Built an Instrument to Prove a Signal Is Actually Used. Agent Memory Has Nothing Like It.",
    "og:description": "A model can contain a decodable signal and not use it to produce its answer.  That sentence is the...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faoyda6qfbkwy9pjc4cn2.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Interpretability Built an Instrument to Prove a Signal Is Actually Used. Agent Memory Has Nothing Like It.",
    "twitter:description": "A model can contain a decodable signal and not use it to produce its answer.  That sentence is the...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faoyda6qfbkwy9pjc4cn2.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://mnemoverse.com/",
      "label": "Mnemoverse",
      "type": "a"
    },
    {
      "href": "https://mnemoverse.com/docs/research/jacobian-lens-explained",
      "label": "original page",
      "type": "a"
    },
    {
      "href": "https://transformer-circuits.pub/2026/workspace",
      "label": "the workspace paper",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/research/global-workspace",
      "label": "Anthropic's overview",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:29Z",
  "source_url": "https://dev.to/izgorodin/interpretability-built-an-instrument-to-prove-a-signal-is-actually-used-agent-memory-has-nothing-2mnf",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 7643,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ff05b9919043/pages/source.html",
  "local_html": "attachments/ff05b9919043/pages/article.html",
  "body_base_path": "attachments/ff05b9919043/pages/",
  "body_markdown_path": "attachments/ff05b9919043/pages/article.md"
}