{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Jason Liu",
  "title_listed": "Compaction Experiments",
  "keywords": "上下文压缩,动量,实验",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://jxnl.co/writing/2025/08/30/context-engineering-compaction/",
  "publish_date": "2025-08",
  "corpus_stage": "推理部署",
  "notes": "上下文压缩实验",
  "id": "22eb87d3d70d",
  "collected_at": "2026-10-02T14:29:26",
  "title": "Two Experiments We Need to Run on AI Agent Compaction - Jason Liu",
  "body_markdown": "What Experiments Should We Run on AI Agent Compaction?\n¶\nTwo core insights:\nIf in-context learning is gradient descent, then compaction is momentum.\nWe can use compaction as a compression system to understand how agents actually behave at scale.\nThis is part of the\nContext Engineering series\n. I'm focusing on compaction because it's where theory meets practice—and where we desperately need empirical research.\nPreviously, through consulting work, I helped companies build better AI systems, and I've been thinking about compaction and how it connects to the research showing that\nin-context learning is gradient descent\n. This theoretical foundation provides a framework for understanding how context management affects agent learning trajectories. If that's true, then compaction is basically a momentum term. And if compaction is momentum, there are two experiments I desperately want to see someone run.\nThis builds on the foundational concepts I've explored in\ncontext engineering\n, where the structure of information flow becomes as critical as the information itself.\nWhat Do These Compaction Terms Mean?\n¶\nCompaction\n: Automatic summarization of conversation history when context windows approach limits, preserving essential information while freeing memory space.\nAgent Trajectory\n: The complete sequence of tool calls, reasoning steps, and responses an agent takes to complete a task. (basically the message array)\nContext Pollution\n: When valuable reasoning context gets flooded with irrelevant information, degrading agent performance. I've written extensively about how this affects AI systems in\nmy analysis of slash commands versus subagents\n.\nMomentum\n: In gradient descent optimization, a component that accelerates convergence by incorporating the direction of previous updates to smooth out oscillations.\nWhy Is Compaction Like Momentum?\n¶\nTraditional gradient descent with momentum:\nθ_{t+1} = θ_t - α∇L(θ_t) + β(θ_t - θ_{t-1})\nConversational learning with compaction:\ncontext_{new} = compact(context_full) + β(learning_trajectory)\nCompaction isn't just storing facts—it's preserving the\nlearned optimization path\n. When you compact \"I tried X, it failed, then Y worked because Z,\" you're maintaining the gradient direction that led to success.\nThis got me thinking: what if we could actually test this? What if we could run experiments that treat compaction as momentum and see what happens?\nExperiment 1: Compaction as Momentum for Long-Running Tasks\n¶\nThe first experiment is about momentum. If compaction preserves learning trajectories, then timing should matter for success rates.\nThe setup\n: Run million-token agent trajectories on complex coding tasks. Test compaction at 50% vs 75% completion vs natural boundaries vs agent-controlled timing.\nThe problem\n: Public benchmarks generally run tasks that are very short and don't burn 700,000 tokens. You need those massive trajectories that only companies like Cursor, Claude Code, or GitHub actually have access to.\nA practical workaround\n: The\nrapid prototyping methodology\nusing Claude Code lets you generate long trajectories for testing. Complex multi-step tasks naturally burn through context as they iterate on tool calls and file modifications. You can study compaction behavior in these prototyping sessions before building production systems.\nThis connects to broader challenges in\nAI engineering communication\n—how do you measure and report progress on systems where the unit of work isn't a feature but a learning trajectory?\nBut we do have examples of long trajectories. Take the\nClaude plays Pokemon\nexperiment—it generates \"enormous amounts of conversation history, far exceeding Claude's 200k context window,\" so they use sophisticated summarization when the conversation history exceeds limits. That's exactly the kind of trajectory where compaction timing would matter.\nKey Metrics\n:\nTask completion success rate\nTime to completion\nNumber of backtracking steps after compaction\nQuality of final deliverable\nResearch Questions\n:\nDoes compaction timing affect success rates?\nCan agents learn to self-compact at optimal moments?\nHow does compaction quality correlate with momentum preservation?\nDoes compaction timing affect how well agents maintain their learning trajectory? Can agents learn to self-compact at optimal moments?\nExperiment 2: Compaction for Trajectory Observability and Population-Level Analysis\n¶\nThe second experiment is more tractable: can we use specialized compaction prompts to understand what's actually happening in agent trajectories?\nBasically, design different compaction prompts for different kinds of analysis:\n1. Failure Mode Detection\nCompact this trajectory focusing on: loops, linter conflicts,\nrecently-deleted code recreation, subprocess errors, and user frustration signals.\n2. Language Switching Analysis\nCompact focusing on: language transitions, framework switches,\ncross-language debugging, and polyglot development patterns.\n3. User Feedback Clustering\nCompact emphasizing: correction requests, preference statements,\nworkflow interruptions, and satisfaction indicators.\nWhat Do We Expect to Discover?\n¶\nI suspect we'd find things like:\n6% of coding trajectories get stuck with linters (I see this constantly in Cursor)\nA bunch of agents recreate code that was just deleted\nExcessive subprocess cycling when language servers act up\nPatterns around when users start giving lots of corrective feedback\nThese failure modes mirror the\ncommon anti-patterns in RAG systems\nbut at the trajectory level rather than the retrieval level.\nHere's why this matters:\nClio\nfound that 10% of Claude conversations are coding-related, which probably influenced building Claude Code. But agent trajectories are totally different from chat conversations. What patterns would we find if we did Clio-style analysis specifically on agent behavior?\nThis type of systematic analysis aligns with the\ndata flywheel approaches\nthat help AI systems improve through user feedback loops—but applied to multi-step reasoning rather than single predictions.\nHow Would the Clustering Approach Work?\n¶\nCompact trajectories\nusing specialized prompts\nCluster compacted summaries\nusing embedding similarity\nIdentify patterns\nacross user bases and use cases\nBuild diagnostic tools\nfor common failure modes\nThis is trajectory-level observability. Instead of just knowing \"agents do coding tasks,\" we could understand \"agents get stuck in linter loops\" or \"agents perform better when users give feedback in this specific way.\"\nIt's similar to the systematic improvement approaches I cover in\nRAG system optimization\n, but focused on agent behavior patterns rather than search relevance.\nWhat Infrastructure Is Missing?\n¶\nContext windows keep getting bigger, but we still hit limits on complex tasks. More importantly, we have no systematic understanding of how agents actually learn and fail over long interactions.\nThis connects to fundamental questions about\nhow AI engineering teams should run standups\n—when your \"product\" is a learning system, traditional software metrics don't capture what matters.\nCompanies building agents could figure out why some trajectories work and others don't. Researchers could connect theory to practice. The field could move beyond single-turn benchmarks toward understanding actual agentic learning.\nHow Do We Get Started?\n¶\nThe momentum experiment realistically needs a company already running coding agents at scale. The observability experiment could work for anyone with substantial agent usage data.\nBoth need access to long trajectories and willingness to run controlled experiments.\nWho Should Run This?\n¶\nThis realistically needs a company already running agents at scale (or a research group with access to long trajectories and willingness to run controlled experiments).\nThe future isn't just about better models. It's about understanding how agents actually learn and optimize over time. Compaction might be the key.\n_This post is part of the\nContext Engineering Series\n. For foundational concepts, start with\nBeyond Chunks: Context Engineering Tool Response\n. To understand how context pollution affects agent performance, read\nSlash Commands vs Subagents\n.\nFor related concepts on AI system evaluation and improvement, explore\nRAG system optimization techniques\nand\nsystematic approaches to AI monitoring\n._",
  "metadata": {
    "title": "Two Experiments We Need to Run on AI Agent Compaction - Jason Liu",
    "description": "If in-context learning is gradient descent, then compaction is momentum. Here are two research directions that could transform how we understand and optimize agentic systems.",
    "keywords": null,
    "author": "Jason Liu",
    "og:type": "website",
    "og:title": "Two Experiments We Need to Run on AI Agent Compaction - Jason Liu",
    "og:description": "If in-context learning is gradient descent, then compaction is momentum. Here are two research directions that could transform how we understand and optimize agentic systems.",
    "og:image": "https://jxnl.co/assets/images/social/writing/2025/08/30/context-engineering-compaction.png",
    "og:image:type": "image/png",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:url": "https://jxnl.co/writing/2025/08/30/context-engineering-compaction/"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/22eb87d3d70d/code/body-000.txt",
      "chars": 44,
      "language": "text"
    },
    {
      "file": "attachments/22eb87d3d70d/code/body-001.txt",
      "chars": 63,
      "language": "text"
    },
    {
      "file": "attachments/22eb87d3d70d/code/body-002.txt",
      "chars": 145,
      "language": "text"
    },
    {
      "file": "attachments/22eb87d3d70d/code/body-003.txt",
      "chars": 124,
      "language": "text"
    },
    {
      "file": "attachments/22eb87d3d70d/code/body-004.txt",
      "chars": 118,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/jxnl/blog/edit/main/docs/writing/posts/context-engineering-compaction.md",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/jxnl/blog/raw/main/docs/writing/posts/context-engineering-compaction.md",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/08/28/context-engineering-index/",
      "label": "Context Engineering series",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.07677",
      "label": "in-context learning is gradient descent",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/08/27/facets-context-engineering/",
      "label": "context engineering",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/08/29/context-engineering-slash-commands-subagents/",
      "label": "my analysis of slash commands versus subagents",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/09/04/context-engineering-rapid-agent-prototyping/",
      "label": "rapid prototyping methodology",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2024/10/15/effective-communication-in-ai-engineering-moving-beyond-vague-updates/",
      "label": "AI engineering communication",
      "type": "a"
    },
    {
      "href": "https://www.lesswrong.com/posts/HyD3khBjnBhvsp8Gb/so-how-well-is-claude-playing-pokemon",
      "label": "Claude plays Pokemon",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/06/11/rag-anti-patterns-with-skylar-payne/",
      "label": "common anti-patterns in RAG systems",
      "type": "a"
    },
    {
      "href": "https://www.anthropic.com/research/clio",
      "label": "Clio",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2024/03/28/data-flywheel/",
      "label": "data flywheel approaches",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2024/08/19/rag-flywheel/",
      "label": "RAG system optimization",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2024/10/25/running-effective-ai-standups/",
      "label": "how AI engineering teams should run standups",
      "type": "a"
    },
    {
      "href": "https://jxnl.co/writing/2025/05/29/systematically-improving-rag-with-raindrop-and-oleve/",
      "label": "systematic approaches to AI monitoring",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:56Z",
  "source_url": "https://jxnl.co/writing/2025/08/30/context-engineering-compaction/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.07677",
      "bytes": 2294443,
      "reused": false,
      "local_path": "papers/repair/99ffb628c4d8104b6d11.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 8351,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/22eb87d3d70d/pages/source.html",
  "local_html": "attachments/22eb87d3d70d/pages/article.html",
  "body_base_path": "attachments/22eb87d3d70d/pages/",
  "body_markdown_path": "attachments/22eb87d3d70d/pages/article.md"
}