{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "苏剑林",
  "title_listed": "Transformer升级之路：13、逆用Leaky ReRoPE",
  "keywords": "Leaky ReRoPE,长度外推",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://spaces.ac.cn/archives/9728",
  "publish_date": "2023-08",
  "corpus_stage": "预训练/SFT",
  "notes": "逆用 Leaky ReRoPE 保持推理速度",
  "id": "d08ef67d3bd0",
  "collected_at": "2026-10-02T15:14:47",
  "title": "Transformer升级之路：13、逆用Leaky ReRoPE - 科学空间|Scientific Spaces",
  "body_markdown": "SEARCH\nMENU\n打赏\n公式\n天象\n链接\n时光\n博览\n归档\nCATEGORIES\n千奇百怪\n天文探索\n数学研究\n物理化学\n信息时代\n生物自然\n图片摄影\n问题百科\n生活/情感\n资源共享\nNEWPOSTS\n如何让GDN2更稳定一些？\n通过微扰分析求解等式约束优化问题\n慢即是快：1. Shampoo的尽...\n排序不等式及其推广\n让炼丹更科学一些（十）：单调性假设...\n让炼丹更科学一些（九）：经典自适应...\n让炼丹更科学一些（八）：多阶段训练...\n动量的新理解：逼近特征层面的梯度下降\n流形上的最速下降：7. Stief...\n除了交叉熵，LM Loss还有什么选择？\nCOMMENTS\ncarl_lei\n: 苏神，之前我一直有一个直觉是：MoE通过不同系数加权得到的FF...\nAli Hatamizadeh\n: Thank you for looking into this...\n苏剑林\n: 对于线性层，取max能得到谱范数，然后可以推出Muon，Muo...\n苏剑林\n: Thanks for sharing! I looked in...\nPurshow\n: 苏老师，请问下取max会不会影响控制下界太宽松了呢\nbookworm\n: 天下苦秦久矣，期待苏老师的教材！\nAli Hatamizadeh\n: Hi ! I am the lead GDN-2 author...\nyixiao chen\n: 谢谢苏神！代码在https://github.com/Chyx...\n苏剑林\n: 欢迎作者！我仔细读了一下，感觉写作还是很清晰的，至少比 htt...\n苏剑林\n: 免训练估计够呛，从实用的角度看，其实也不应该相信任何免训练的可能\nUSERLOGIN\n登录\n科学空间|Scientific Spaces\n登录\n打赏\n公式\n天象\n链接\n时光\n博览\n归档\n渴望成为一个小飞侠\n欢迎订阅\n个性邮箱\n天象信息\n观测ISS\nLaTeX\n关于博主\n欢迎访问“科学空间”，这里将与您共同探讨自然科学，回味人生百态；也期待大家的分享～\n千奇百怪\nEverything\n天文探索\nAstronomy\n数学研究\nMathematics\n物理化学\nPhy-chem\n信息时代\nBig-Data\n生物自然\nBiology\n图片摄影\nPhotograph\n问题百科\nQuestions\n生活/情感\nLife-Feeling\n资源共享\nResources\n千奇百怪\n天文探索\n数学研究\n物理化学\n信息时代\n生物自然\n图片摄影\n问题百科\n生活/情感\n资源共享\n首页\n信息时代\nTransformer升级之路：13、逆用Leaky ReRoPE\n14\nAug\nTransformer升级之路：13、逆用Leaky ReRoPE\nBy\n苏剑林\n|\n2023-08-14\n|\n38699位读者\n|\n:\n上周在\n《Transformer升级之路：12、无限外推的ReRoPE？》\n中，笔者提出了ReRoPE和Leaky ReRoPE，诸多实验结果表明，它们能够在几乎不损失训练效果的情况下免微调地扩展LLM的Context长度，并且实现了“longer context, lower loss”的理想特性，此外跟NTK-aware Scaled RoPE不同的是，其中ReRoPE似乎还有表现出了无限的Context处理能力。\n总之，ReRoPE看起来相当让人满意，但美中不足的是会增加推理成本，具体表现为第一步推理需要算两次Attention，以及后续每步推理需要重新计算位置编码。本文试图通过在训练中逆用Leaky ReRoPE的方法来解决这个问题。\n回顾\n#\n让我们不厌其烦地重温一下：RoPE形式上是一种绝对位置编码，但实际达到的效果是相对位置编码，对应的相对位置矩阵是：\n\\begin{equation}\\begin{pmatrix}0 & \\\\\n1 & 0 & \\\\\n2 & 1 & 0 &\\\\\n3 & 2 & 1 & 0 & \\\\\n\\ddots & 3 & 2 & 1 & 0 & \\\\\n\\ddots & \\ddots & 3 & 2 & 1 & 0 & \\\\\n\\ddots & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots \\\\\n\\small{L - 2} & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots \\\\\n\\small{L - 1} & \\small{L - 2} & \\ddots & \\ddots & \\ddots & 3 & 2 & 1 & 0 & \\\\\n\\end{pmatrix}\\label{eq:rope}\\end{equation}\n为了在保留局域性的同时避免Long Context导致位置越界问题，Leaky ReRoPE将推理阶段的相对位置矩阵改为：\n\\begin{equation}\\begin{pmatrix}\n\\color{red}{0} & \\\\\n\\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\small{w + \\frac{1}{k}}} & \\color{green}{w} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\small{w + \\frac{2}{k}}} & \\color{green}{\\small{w + \\frac{1}{k}}} & \\color{green}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\ddots} & \\color{green}{\\small{w + \\frac{2}{k}}} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\\\\n\\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\small{w + \\frac{2}{k}}} & \\color{green}{\\small{w + \\frac{1}{k}}} & \\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\small{w + \\frac{L-1-w}{k}}} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\small{w + \\frac{2}{k}}} & \\color{green}{\\small{w + \\frac{1}{k}}} & \\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\end{pmatrix}\\label{eq:leaky-rerope}\\end{equation}\n其中$w$是窗口宽度，大概取训练长度的$\\frac{1}{4}$到$\\frac{1}{2}$，$k$用来调节可处理的最大长度，一般使得$w + \\frac{L-1-w}{k}$不超过训练长度的一半为佳。至于ReRoPE，则是直接取了$k\\to\\infty$的极限：\n\\begin{equation}\\begin{pmatrix}\n\\color{red}{0} & \\\\\n\\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{w} & \\color{green}{w} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{w} & \\color{green}{w} & \\color{green}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\ddots} & \\color{green}{w} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\\\\n\\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{w} & \\color{green}{w} & \\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{w} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{w} & \\color{green}{w} & \\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{2} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\end{pmatrix}\\label{eq:rerope}\\end{equation}\n反转\n#\n从上一篇的评测结果来看，作为一种免训练的外推方案，ReRoPE和Leaky ReRoPE的效果都是相当让人满意的，既没有损失训练长度内的效果，又实现了“Longer Context, Lower Loss”。唯一美中不足的是，它们的推理速度相比原本的Attention来说是变慢的，并且目前尚不兼容Flash Attention等加速技术。\n那么，能否反过来呢？ReRoPE/Leaky ReRoPE在训练阶段是正常速度的RoPE，推理阶段则是变慢了，反过来也就是说：能否让训练阶段变慢，让推理阶段变为常规的RoPE？可能有读者疑惑：为什么会想要让训练阶段变慢？训练成本不是更高吗？这是因为ReRoPE/Leaky ReRoPE是一种长度外推方法，场景是“\nTrain Short, Test Long\n”，训练速度的变慢是短期的、可控的，推理速度的变慢才是长期的、难顶的，所以相较之下，如果是同等程度的变慢的话，我们更愿意将变慢的部分放到训练阶段。\n让我们再看一下Leaky ReRoPE，它在训练阶段的相对位置矩阵是步长为1的式$\\eqref{eq:rope}$，推理阶段则在$w$的窗口内使用$1$的步长，在窗口外使用$\\frac{1}{k} < 1$的步长，即式$\\eqref{eq:leaky-rerope}$，换句话说，\n差别是推理阶段窗口外使用更小的步长\n。如果我们反过来，在训练阶段使用Leaky ReRoPE，并让它窗口外的步长大于$1$，那么按照“推理阶段窗口外使用更小的步长”的原则，推理阶段窗口外是否就可以使用等于$1$的步长，从而退化为RoPE了？\n笔者将以上想法称之为“InvLeaky ReRoPE（Inverse Leaky ReRoPE）”。事不宜迟，我们马上做实验测试。\n实验\n#\n继续之前的“\nGAU\n+\nDeep Norm\n+\nTiger\n+ 语言模型”实验组合，在训练阶段使用$k=1/16, w=128$的Leaky ReRoPE，在推理阶段使用正常的RoPE，测试结果如下：\n\\begin{array}{c|cc}\n\\hline\n\\text{测试长度} & 512(\\text{训练}) & 4096(\\text{重复}) & 4096(\\text{不重复})\\\\\n\\hline\n\\text{Baseline} & 49.41\\% & 24.17\\% & 23.16\\% \\\\\n\\text{Baseline-}\\log n & 49.40\\% & 24.60\\% & 24.02\\% \\\\\n\\hline\n\\text{NTK-RoPE-fixed} & 49.41\\% & 51.86\\% & 39.61\\% \\\\\n\\text{NTK-RoPE-}\\log n^{\\color{red}{\\dagger}}\\text{-fixed} & 49.41\\% & 55.94\\% & 41.11\\% \\\\\n\\text{NTK-RoPE-}\\log n\\text{-fixed} & 49.40\\% & 62.85\\% & 44.14\\% \\\\\n\\text{NTK-RoPE-mixed} & 49.41\\% & 53.09\\% & 40.12\\% \\\\\n\\text{NTK-RoPE-}\\log n^{\\color{red}{\\dagger}}\\text{-mixed} & 49.41\\% & 59.11\\% & 42.38\\% \\\\\n\\text{NTK-RoPE-}\\log n\\text{-mixed} & 49.40\\% & 68.91\\% & 45.41\\% \\\\\n\\hline\n\\text{ReRoPE-w256} & 49.41\\% & 77.90\\% & 48.48\\% \\\\\n\\text{ReRoPE-w256-}\\log n^{\\color{red}{\\dagger}} & 49.41\\% & 82.40\\% & 48.85\\% \\\\\n\\text{ReRoPE-w256-}\\log n & 49.40\\% & \\boldsymbol{85.12\\%} & \\boldsymbol{49.07\\%} \\\\\n\\hline\n\\text{InvLeaky ReRoPE-w128-}\\log n & 49.38\\% & 82.25\\% & 48.32\\% \\\\\n\\text{InvLeaky ReRoPE-w128-b8-}\\log n & 49.62\\% & 81.15\\% & 48.85\\% \\\\\n\\hline\n\\text{HFWA} & 48.70\\% & 80.84\\% & 48.15\\% \\\\\n\\hline\n\\end{array}\n其中$\\text{b8}$是指RoPE的频率底数从10000换成了80000。可以看到，“Leaky ReRoPE → RoPE”的InvLeaky ReRoPE虽然效果上不如“RoPE → ReRoPE/Leaky ReRoPE”，但依然胜过了HFWA，并且由于推理阶段是常规的RoPE，可以套用现成的加速技术，因此依然是有相当竞争力的。此外，笔者对$k,w,b$等参数做了一些简单的调参，发现最优解基本上就是以上两个组合了，即“\n$k$设置为‘扩展倍数的2倍的倒数’、$w$设置为训练长度的$\\frac{1}{4}$、$b$可选乘以扩展倍数\n”。\n那么，InvLeaky ReRoPE对训练速度有多大影响呢？在上述实验中，模型是1亿参数量，训练长度是512，每1000步的训练时间从330秒增加到了350秒，增加不到10%，当然这里边有GAU的原因，因为GAU是单头的注意力，本就比多头注意力快。如果多头注意力或者训练长度更长的话，增加幅度应该会大一些，但目测应该不超过50%都是可以接受的。\n小结\n#\n本文提出了Leaky ReRoPE的“逆用”做法，通过在训练阶段使用更大步长的Leaky ReRoPE，使得推理阶段可以退回常规的RoPE，从而可以保持推理速度不变，实验结果显示这种做法还是有一定的竞争力的。\n转载到请包括本文地址：\nhttps://spaces.ac.cn/archives/9728\n更详细的转载事宜请参考：\n《科学空间FAQ》\n如果您还有什么疑惑或建议，欢迎在下方评论区继续讨论。\n如果您觉得本文还不错，欢迎\n分享\n/\n打赏\n本文。打赏并非要从中获得收益，而是希望知道科学空间获得了多少读者的真心关注。当然，如果你无视它，也不会影响你的阅读。再次表示欢迎和感谢！\n打赏\n微信打赏\n支付宝打赏\n因为网站后台对打赏并无记录，因此欢迎在打赏时候备注留言。\n你还可以\n点击这里\n或在下方评论区留言来告知你的建议或需求。\n如果您需要引用本文，请参考：\n苏剑林. (Aug. 14, 2023). 《Transformer升级之路：13、逆用Leaky ReRoPE 》[Blog post]. Retrieved from\nhttps://spaces.ac.cn/archives/9728\n@online{kexuefm-9728,\ntitle={Transformer升级之路：13、逆用Leaky ReRoPE},\nauthor={苏剑林},\nyear={2023},\nmonth={Aug},\nurl={\\url{https://spaces.ac.cn/archives/9728}},\n}\n分类：\n信息时代\n标签：\nattention\n,\n位置编码\n,\n泛化\n,\n外推\n,\nrope\n6 评论\n<\nTransformer升级之路：12、无限外推的ReRoPE？\n|\nTransformer升级之路：14、当HWFA遇见ReRoPE\n>\n你也许还对下面的内容感兴趣\n如何让GDN2更稳定一些？\n简单谈谈K3的MoE和Attention\n将Softmax Attention线性化为Gated DeltaNet\nLogSumExp和Softmax的泰勒展开\nAttention Residuals 回忆录\nDeltaNet的核心逆矩阵的元素总是在[-1, 1]内\n为什么DeltaNet要加L2 Normalize？\n低精度Attention可能存在有偏的舍入误差\n为什么线性注意力要加Short Conv？\nQK-Clip：让Muon在Scaleup之路上更进一步\n发表你的看法\nTransformer升级之路：13、逆用Leaky ReRoPE R11; AI 資訊\nAugust 14th, 2023\n[...]​Read More [...]\n回复评论\ndenghao li\nAugust 14th, 2023\n苏神这个“难顶的”一词成功阻止了大学生们直接复制粘贴进自己论文里^。^\n回复评论\npan\n发表于\n                August 29th, 2023\n我们做了个基于AI大模型的自动校对工具：\nhttp://zyinfo.pro/ai/review/\n以后这类小错误估计都能查出来。\n其实也可以通过AI 重写一部分？\n回复评论\n苏剑林\n发表于\n                August 31st, 2023\n谢谢推荐，有机会试用一下。\n不过“难顶的”不是错误，而是故意口语化罢了（或者说白话化）。\n回复评论\nMingyang Deng\nJanuary 2nd, 2024\n苏老师您好，今天读到您的博客，很受启发。关于速度问题，ReRope有没有可能这样实现呢：\n(1) 将输入划分成大小是w/2的batch\n(2) 每个batch和自己及上一个batch采用RoPe计算attention；和再之前的batch使用w计算attention\n这样推理的开销应该是一样的（每两个batch之间只计算了一次attention），同时效果几乎是ReRope（除了有一些w/2到w之间的值被近似成了w）\n回复评论\n苏剑林\n发表于\n                January 5th, 2024\n这正是flash attention的分块思路！已经有读者提供过一个参考实现了\nhttps://kexue.fm/archives/9708/comment-page-2#comment-22614\n回复评论\n取消回复\n你的大名\n电子邮箱\n个人网站（选填）\n1. 可以使用LaTeX代码，点击“预览效果”可查看效果；\n2. 可以通过点击评论楼层编号来引用该楼层；\n3. 网站可能会有点卡，如非确认评论失败，请\n不要重复点击提交\n。\n内容速览\n回顾\n反转\n实验\n小结\n智能搜索\n支持整句搜索！网站自动使用\n结巴分词\n进行分词，并结合ngrams排序算法给出合理的搜索结果。\n热门标签\nattention\n生成模型\n优化\n模型\n语言模型\n梯度\n优化器\n矩阵\n概率\n网站\n转载\n微分方程\n分析\n天象\n深度学习\n积分\npython\n几何\n扩散\n力学\n无监督\n损失函数\n节日\n线性\n生活\n随机文章\n中位数（Median）简介\n用傅里叶级数拟合一维概率密度函数\nbert4keras在手，baseline我有：百度LIC2020\n日本科学家发现人体能发出可见光(图)\n闲聊：神经网络与深度学习\n【NASA每日一图】明亮的超新星爆发\n人不能忘本|我的数学竞赛题\n《自然极值》系列——1.前言\n傅里叶变换：只需要异想天开？\n基于遗忘假设的平滑公式\n最近评论\ncarl_lei\n: 苏神，之前我一直有一个直觉是：MoE通过不同系数加权得到的FFN相比原始的大FFN（也就是直接...\nAli Hatamizadeh\n: Thank you for looking into this and featuring i...\n苏剑林\n: 对于线性层，取max能得到谱范数，然后可以推出Muon，Muon比Adam、SGD等要好，所以...\n苏剑林\n: Thanks for sharing! I looked into this improvem...\nPurshow\n: 苏老师，请问下取max会不会影响控制下界太宽松了呢\nbookworm\n: 天下苦秦久矣，期待苏老师的教材！\nAli Hatamizadeh\n: Hi ! I am the lead GDN-2 author.We've addressed...\nyixiao chen\n: 谢谢苏神！代码在https://github.com/Chyxx/CyclicFlowAtte...\n苏剑林\n: 欢迎作者！我仔细读了一下，感觉写作还是很清晰的，至少比 https://papers.cool...\n苏剑林\n: 免训练估计够呛，从实用的角度看，其实也不应该相信任何免训练的可能\n友情链接\nCool Papers\n数学研发\nSeatop\nXiaoxia\n积分表-网络版\n丝路博傲\n数学之家\n有趣天文奇观\nTwistedW\ngodweiyang\nAI柠檬\n王登科-DK博客\nESON\n枫之羽\ncoding-zuo\n博科园\n孔皮皮的博客\n运鹏的博客\njiming.site\nOmegaXYZ\nEAI猩球\n文举的博客\n汇尘轩\n申请链接\n本站采用创作共用版权协议，要求署名、非商业用途和保持一致。转载本站内容必须也遵循“\n署名-非商业用途-保持一致\n”的创作共用协议。\n© 2009-2026 Scientific Spaces. All rights reserved. Theme by\nlaogui\n. Powered by\nTypecho\n. 备案号:\n粤ICP备09093259号-1/2\n。",
  "metadata": {
    "title": "Transformer升级之路：13、逆用Leaky ReRoPE - 科学空间|Scientific Spaces",
    "description": "上周在《Transformer升级之路：12、无限外推的ReRoPE？》中，笔者提出了ReRoPE和Leaky ReRoPE，诸多实验结果表明，它们能够在几乎不损失训练效果的情况下免微调地扩展L...",
    "keywords": "attention,位置编码,泛化,外推,rope",
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/images/slide-button.png",
      "bytes": 4328,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-fcd8b7d1212b1f5eab2b.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/images/slide-button.png",
      "alt": "MobileSideBar",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/images/rss.png",
      "bytes": 19283,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-043e6308dc873ff9fa67.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/images/rss.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/images/mail.png",
      "bytes": 16811,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-942ff5c7b650f4d3d31f.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/images/mail.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/images/Saturn.png",
      "bytes": 2524,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-53b9d00850633d612de3.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/images/Saturn.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/images/iss.png",
      "bytes": 3301,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-896b482bdde5ec1df88a.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/images/iss.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/images/pi.png",
      "bytes": 939,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-3f7f2f2318249e4eac9a.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/images/pi.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/images/mlogo.png",
      "bytes": 18685,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-cbbace78f2cd4551d156.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/images/mlogo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/payment/wx.png",
      "bytes": 36540,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-c5d79f289db1f5d431bc.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/payment/wx.png",
      "alt": "科学空间",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://spaces.ac.cn/usr/themes/geekg/payment/zfb.png",
      "bytes": 31565,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-4345e58697a529a3d3a5.png",
      "src": "https://spaces.ac.cn/usr/themes/geekg/payment/zfb.png",
      "alt": "科学空间",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://kexue.fm/usr/themes/geekg/images/cc.gif",
      "bytes": 2031,
      "reused": false,
      "local_path": "attachments/d08ef67d3bd0/images/body-b2904ced8e4508d61bed.gif",
      "src": "https://kexue.fm/usr/themes/geekg/images/cc.gif",
      "alt": "署名-非商业用途-保持一致",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://spaces.ac.cn/reward.html",
      "label": "打赏",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/latex.html",
      "label": "公式",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/ac.html",
      "label": "天象",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/links.html",
      "label": "链接",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/me.html",
      "label": "时光",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/science.html",
      "label": "博览",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/content.html",
      "label": "归档",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Everything",
      "label": "千奇百怪",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Astronomy",
      "label": "天文探索",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Mathematics",
      "label": "数学研究",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Phy-chem",
      "label": "物理化学",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Big-Data",
      "label": "信息时代",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Biology",
      "label": "生物自然",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Photograph",
      "label": "图片摄影",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Questions",
      "label": "问题百科",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Life-Feeling",
      "label": "生活/情感",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/category/Resources",
      "label": "资源共享",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11945",
      "label": "如何让GDN2更稳定一些？",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11928",
      "label": "通过微扰分析求解等式约束优化问题",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11917",
      "label": "慢即是快：1. Shampoo的尽...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11910",
      "label": "排序不等式及其推广",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11885",
      "label": "让炼丹更科学一些（十）：单调性假设...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11882",
      "label": "让炼丹更科学一些（九）：经典自适应...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11879",
      "label": "让炼丹更科学一些（八）：多阶段训练...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11875",
      "label": "动量的新理解：逼近特征层面的梯度下降",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11864",
      "label": "流形上的最速下降：7. Stief...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11854",
      "label": "除了交叉熵，LM Loss还有什么选择？",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/10699/comment-page-3",
      "label": "carl_lei : 苏神，之前我一直有一个直觉是：MoE通过不同系数加权得到的FF...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11945/comment-page-1",
      "label": "Ali Hatamizadeh : Thank you for looking into this...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11340/comment-page-1",
      "label": "苏剑林 : 对于线性层，取max能得到谱范数，然后可以推出Muon，Muo...",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/1328/comment-page-1",
      "label": "bookworm : 天下苦秦久矣，期待苏老师的教材！",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11848/comment-page-2",
      "label": "苏剑林 : 免训练估计够呛，从实用的角度看，其实也不应该相信任何免训练的可能",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/admin/login.php",
      "label": "登录",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/",
      "label": "科学空间|Scientific Spaces",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/feed",
      "label": "欢迎订阅",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/119",
      "label": "个性邮箱",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/41",
      "label": "观测ISS",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9728",
      "label": "Transformer升级之路：13、逆用Leaky ReRoPE",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9708",
      "label": "《Transformer升级之路：12、无限外推的ReRoPE？》",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/8934",
      "label": "GAU",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/8978",
      "label": "Deep Norm",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9512",
      "label": "Tiger",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/6508",
      "label": "《科学空间FAQ》",
      "type": "a"
    },
    {
      "href": "http://mail.qq.com/cgi-bin/qm_share?t=qm_mailme&email=tN7d1drY3drrx8H0xcWa19vZ",
      "label": "点击这里",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/attention/",
      "label": "attention",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81/",
      "label": "位置编码",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%B3%9B%E5%8C%96/",
      "label": "泛化",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E5%A4%96%E6%8E%A8/",
      "label": "外推",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/rope/",
      "label": "rope",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9731",
      "label": "Transformer升级之路：14、当HWFA遇见ReRoPE",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11848",
      "label": "简单谈谈K3的MoE和Attention",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11823",
      "label": "将Softmax Attention线性化为Gated DeltaNet",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11814",
      "label": "LogSumExp和Softmax的泰勒展开",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11664",
      "label": "Attention Residuals 回忆录",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11563",
      "label": "DeltaNet的核心逆矩阵的元素总是在[-1, 1]内",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11486",
      "label": "为什么DeltaNet要加L2 Normalize？",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11371",
      "label": "低精度Attention可能存在有偏的舍入误差",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11320",
      "label": "为什么线性注意力要加Short Conv？",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11126",
      "label": "QK-Clip：让Muon在Scaleup之路上更进一步",
      "type": "a"
    },
    {
      "href": "https://news.aitime.space/2023/08/23209/",
      "label": "Transformer升级之路：13、逆用Leaky ReRoPE R11; AI 資訊",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9728/comment-page-1?replyTo=22497",
      "label": "回复评论",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9728/comment-page-1?replyTo=22501",
      "label": "回复评论",
      "type": "a"
    },
    {
      "href": "http://zyinfo.pro/",
      "label": "pan",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9728/comment-page-1?replyTo=22604",
      "label": "回复评论",
      "type": "a"
    },
    {
      "href": "https://kexue.fm/",
      "label": "苏剑林",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9728/comment-page-1?replyTo=22619",
      "label": "回复评论",
      "type": "a"
    },
    {
      "href": "http://lambertae.github.io/",
      "label": "Mingyang Deng",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9728/comment-page-1?replyTo=23431",
      "label": "回复评论",
      "type": "a"
    },
    {
      "href": "https://kexue.fm/archives/9708/comment-page-2",
      "label": "https://kexue.fm/archives/9708/comment-page-2#comment-22614",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/9728/comment-page-1?replyTo=23453",
      "label": "回复评论",
      "type": "a"
    },
    {
      "href": "https://github.com/fxsjy/jieba",
      "label": "结巴分词",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E7%94%9F%E6%88%90%E6%A8%A1%E5%9E%8B/",
      "label": "生成模型",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E4%BC%98%E5%8C%96/",
      "label": "优化",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%A8%A1%E5%9E%8B/",
      "label": "模型",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B/",
      "label": "语言模型",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%A2%AF%E5%BA%A6/",
      "label": "梯度",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E4%BC%98%E5%8C%96%E5%99%A8/",
      "label": "优化器",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E7%9F%A9%E9%98%B5/",
      "label": "矩阵",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%A6%82%E7%8E%87/",
      "label": "概率",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E7%BD%91%E7%AB%99/",
      "label": "网站",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E8%BD%AC%E8%BD%BD/",
      "label": "转载",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E5%BE%AE%E5%88%86%E6%96%B9%E7%A8%8B/",
      "label": "微分方程",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E5%88%86%E6%9E%90/",
      "label": "分析",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E5%A4%A9%E8%B1%A1/",
      "label": "天象",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/",
      "label": "深度学习",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E7%A7%AF%E5%88%86/",
      "label": "积分",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/python/",
      "label": "python",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E5%87%A0%E4%BD%95/",
      "label": "几何",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%89%A9%E6%95%A3/",
      "label": "扩散",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E5%8A%9B%E5%AD%A6/",
      "label": "力学",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%97%A0%E7%9B%91%E7%9D%A3/",
      "label": "无监督",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0/",
      "label": "损失函数",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E8%8A%82%E6%97%A5/",
      "label": "节日",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E7%BA%BF%E6%80%A7/",
      "label": "线性",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/tag/%E7%94%9F%E6%B4%BB/",
      "label": "生活",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/11693",
      "label": "中位数（Median）简介",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/10007",
      "label": "用傅里叶级数拟合一维概率密度函数",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/7321",
      "label": "bert4keras在手，baseline我有：百度LIC2020",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/33",
      "label": "日本科学家发现人体能发出可见光(图)",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/3331",
      "label": "闲聊：神经网络与深度学习",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/48",
      "label": "【NASA每日一图】明亮的超新星爆发",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/272",
      "label": "人不能忘本|我的数学竞赛题",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/1065",
      "label": "《自然极值》系列——1.前言",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/2555",
      "label": "傅里叶变换：只需要异想天开？",
      "type": "a"
    },
    {
      "href": "https://spaces.ac.cn/archives/4182",
      "label": "基于遗忘假设的平滑公式",
      "type": "a"
    },
    {
      "href": "https://papers.cool/",
      "label": "Cool Papers",
      "type": "a"
    },
    {
      "href": "https://bbs.emath.ac.cn/",
      "label": "数学研发",
      "type": "a"
    },
    {
      "href": "http://www.seatop.com.cn/",
      "label": "Seatop",
      "type": "a"
    },
    {
      "href": "https://xiaoxia.org/",
      "label": "Xiaoxia",
      "type": "a"
    },
    {
      "href": "https://kexue.fm/sci/integral/index.html",
      "label": "积分表-网络版",
      "type": "a"
    },
    {
      "href": "http://blog.dvxj.com/",
      "label": "丝路博傲",
      "type": "a"
    },
    {
      "href": "http://www.2math.cn/",
      "label": "数学之家",
      "type": "a"
    },
    {
      "href": "http://interesting-sky.china-vo.org/",
      "label": "有趣天文奇观",
      "type": "a"
    },
    {
      "href": "http://www.twistedwg.com/",
      "label": "TwistedW",
      "type": "a"
    },
    {
      "href": "https://godweiyang.com/",
      "label": "godweiyang",
      "type": "a"
    },
    {
      "href": "https://blog.ailemon.net/",
      "label": "AI柠檬",
      "type": "a"
    },
    {
      "href": "https://greatdk.com/",
      "label": "王登科-DK博客",
      "type": "a"
    },
    {
      "href": "https://blog.eson.org/",
      "label": "ESON",
      "type": "a"
    },
    {
      "href": "https://fzhiy.net/",
      "label": "枫之羽",
      "type": "a"
    },
    {
      "href": "https://coding-zuo.github.io/",
      "label": "coding-zuo",
      "type": "a"
    },
    {
      "href": "https://www.bokeyuan.net/",
      "label": "博科园",
      "type": "a"
    },
    {
      "href": "https://www.kppkkp.top/",
      "label": "孔皮皮的博客",
      "type": "a"
    },
    {
      "href": "https://yunpengtai.top/",
      "label": "运鹏的博客",
      "type": "a"
    },
    {
      "href": "https://jiming.site/",
      "label": "jiming.site",
      "type": "a"
    },
    {
      "href": "https://www.omegaxyz.com/",
      "label": "OmegaXYZ",
      "type": "a"
    },
    {
      "href": "https://www.robotech.ink/",
      "label": "EAI猩球",
      "type": "a"
    },
    {
      "href": "https://liwenju0.com/",
      "label": "文举的博客",
      "type": "a"
    },
    {
      "href": "https://kirigaya.cn/",
      "label": "汇尘轩",
      "type": "a"
    },
    {
      "href": "https://kexue.fm/links.html",
      "label": "申请链接",
      "type": "a"
    },
    {
      "href": "http://creativecommons.org/licenses/by-nc-nd/2.5/cn/",
      "label": "",
      "type": "a"
    },
    {
      "href": "http://www.laogui.com/",
      "label": "laogui",
      "type": "a"
    },
    {
      "href": "http://typecho.org/",
      "label": "Typecho",
      "type": "a"
    },
    {
      "href": "https://beian.miit.gov.cn/",
      "label": "粤ICP备09093259号-1/2",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:53:05Z",
  "source_url": "https://spaces.ac.cn/archives/9728",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11393,
  "expected_body_images": 10,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/d08ef67d3bd0/pages/source.html",
  "local_html": "attachments/d08ef67d3bd0/pages/article.html",
  "body_base_path": "attachments/d08ef67d3bd0/pages/",
  "body_markdown_path": "attachments/d08ef67d3bd0/pages/article.md"
}