{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Arun Kumar",
  "title_listed": "\"RoPE extrapolates to longer contexts\" — someone finally measured it, and it doesn't",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/narotra05hp/rope-extrapolates-to-longer-contexts-someone-finally-measured-it-and-it-doesnt-5dnk",
  "publish_date": "2026-09-10",
  "corpus_stage": "",
  "notes": "",
  "id": "232c10c187bb",
  "collected_at": "2026-10-02T13:54:54",
  "title": "\"RoPE extrapolates to longer contexts\" — someone finally measured it, and it doesn't - DEV Community",
  "body_markdown": "You've read this sentence, or something close to it:\nRoPE handles longer sequences than absolute positional encoding because rotations are periodic.\nI believed it too. Then I went looking for where it was measured, and found the opposite.\nThe claim has a four-paper history, and following it in order is more interesting than the claim itself.\n2017: a hedge\nThe original Transformer paper picks sinusoidal encodings over learned ones, and gives a reason:\nWe chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training.\nMay.\nVaswani and colleagues were careful. It's a hypothesis, offered as a tiebreaker between two options that — by their own testing — \"produced nearly identical results\".\nThat word survives almost nowhere downstream. It got quoted into oblivion as a property.\n2021: RoPE arrives\nSu and colleagues introduce rotary embeddings and list \"the flexibility of sequence length\" among its properties.\nNothing wrong with that. It's a design claim about the mechanism, made before anyone had measured the ceiling. But between 2017's\nmay\nand 2021's property list, \"might extrapolate\" quietly became \"extrapolates\".\n2022: someone measures it\nThis is the paper that should have ended the folklore, and mostly didn't.\nThe ALiBi paper (Press, Smith and Lewis, ICLR 2022) exists to test length extrapolation, and it tests the incumbents first. On sinusoidal encodings — the thing the 2017 paper hoped would extrapolate — the verdict is that it \"in practice has very limited extrapolation capabilities\".\nOn rotary:\nThough the rotary position method improves over the sinusoidal one, it still does not achieve satisfying results.\nBetter than sinusoids. Still not good. And they put numbers on it: a model trained at length 512 keeps improving perplexity for about\n200 extra tokens\n. At length 1024, about\n100\n.\nTwo hundred tokens. Not 2x, not 10x. And the paper notes that even this \"comes at the cost of slower training and inference\".\n2023: the patch that proves the point\nIf RoPE extrapolated, Position Interpolation wouldn't need to exist.\nChen and colleagues open by naming the problem directly — many pre-trained LLMs, LLaMA included, \"use positional encodings that have weak extrapolation properties\" — and RoPE is the example they name. Their fix doesn't extrapolate at all. It\ninterpolates\n: squeeze the position indices down into the range the model already saw.\nWhy not just push past the trained length? Because doing so \"may lead to catastrophically high attention scores that completely ruin the self-attention mechanism\".\nThat's the strongest evidence available. A whole technique exists because the free-extrapolation story isn't true.\nThe other thing everyone gets backwards\nWhile I was in the RoFormer paper I hit a second surprise. People describe RoPE as\nrelative\npositional encoding, in contrast to\nabsolute\n. Here's the abstract:\nthe proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation\nIt's both. Simultaneously. And once you see the mechanism it's obvious why: each vector rotates by an angle set by its own\nabsolute\nindex, and when you take the dot product of a rotated query and a rotated key, the absolute angles cancel and only the\noffset\nsurvives.\nAbsolute goes in. Relative comes out. One operation.\nThe 2017 side wasn't purely absolute either, incidentally — the sinusoidal choice was motivated by the hope the model would \"easily learn to attend by relative positions\". The clean absolute/relative split was never real.\nWhat I'd actually keep\nNone of this makes RoPE a bad choice. It dominates modern stacks for good reasons — it composes with the KV cache, it never touches the value vector, and it has a provable long-term decay property that sinusoids never claimed.\nBut three corrections stuck with me:\nIt doesn't extrapolate for free.\nMeasured at ~200 extra tokens past a 512 training length. If you need a longer window you need Position Interpolation, or YaRN, or retraining. The encoding alone doesn't give it to you.\nIt isn't \"relative instead of absolute\".\nIt's both, by construction.\nThe 2017 authors weren't wrong.\nThey wrote\nmay\n, and hedged correctly. The folklore is downstream of people dropping the hedge.\nLonger version with every quote sourced at\ndiffstudy.com\n. Papers:\n1706.03762\n(Transformer),\n2104.09864\n(RoFormer),\n2108.12409\n(ALiBi),\n2306.15595\n(Position Interpolation).",
  "metadata": {
    "title": "\"RoPE extrapolates to longer contexts\" — someone finally measured it, and it doesn't - DEV Community",
    "description": "You've read this sentence, or something close to it: RoPE handles longer sequences than absolute... Tagged with ai, machinelearning, llm, computerscience.",
    "keywords": "ai, machinelearning, llm, computerscience, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/narotra05hp/rope-extrapolates-to-longer-contexts-someone-finally-measured-it-and-it-doesnt-5dnk",
    "og:title": "\"RoPE extrapolates to longer contexts\" — someone finally measured it, and it doesn't",
    "og:description": "You've read this sentence, or something close to it: RoPE handles longer sequences than absolute...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fun014hv267oxt2qilinj.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "\"RoPE extrapolates to longer contexts\" — someone finally measured it, and it doesn't",
    "twitter:description": "You've read this sentence, or something close to it: RoPE handles longer sequences than absolute...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fun014hv267oxt2qilinj.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://diffstudy.com/rope-vs-absolute-positional-encoding/",
      "label": "diffstudy.com",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03762",
      "label": "1706.03762",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2104.09864",
      "label": "2104.09864",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2108.12409",
      "label": "2108.12409",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2306.15595",
      "label": "2306.15595",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:01Z",
  "source_url": "https://dev.to/narotra05hp/rope-extrapolates-to-longer-contexts-someone-finally-measured-it-and-it-doesnt-5dnk",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03762",
      "bytes": 2215244,
      "reused": true,
      "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2104.09864",
      "bytes": 599289,
      "reused": false,
      "local_path": "papers/repair/b64aa8c5e278403b64ef.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2108.12409",
      "bytes": 759030,
      "reused": false,
      "local_path": "papers/repair/238b286ab4f338940590.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2306.15595",
      "bytes": 751217,
      "reused": false,
      "local_path": "papers/repair/e4d0dc2cbdf1c95e349e.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 4526,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/232c10c187bb/pages/source.html",
  "local_html": "attachments/232c10c187bb/pages/article.html",
  "body_base_path": "attachments/232c10c187bb/pages/",
  "body_markdown_path": "attachments/232c10c187bb/pages/article.md"
}