{ "source_file": "0930", "source_platform": "个人博客", "author": "苏剑林", "title_listed": "Transformer升级之路：14、当HWFA遇见ReRoPE", "keywords": "HWFA,ReRoPE,长度外推", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://spaces.ac.cn/archives/9731", "publish_date": "2023-08", "corpus_stage": "预训练/SFT", "notes": "HWFA 与 ReRoPE 组合", "id": "f74c12ee5ad8", "collected_at": "2026-10-02T15:14:48", "title": "Transformer升级之路：14、当HWFA遇见ReRoPE - 科学空间|Scientific Spaces", "body_markdown": "SEARCH\nMENU\n打赏\n公式\n天象\n链接\n时光\n博览\n归档\nCATEGORIES\n千奇百怪\n天文探索\n数学研究\n物理化学\n信息时代\n生物自然\n图片摄影\n问题百科\n生活/情感\n资源共享\nNEWPOSTS\n如何让GDN2更稳定一些？\n通过微扰分析求解等式约束优化问题\n慢即是快：1. Shampoo的尽...\n排序不等式及其推广\n让炼丹更科学一些（十）：单调性假设...\n让炼丹更科学一些（九）：经典自适应...\n让炼丹更科学一些（八）：多阶段训练...\n动量的新理解：逼近特征层面的梯度下降\n流形上的最速下降：7. Stief...\n除了交叉熵，LM Loss还有什么选择？\nCOMMENTS\ncarl_lei\n: 苏神，之前我一直有一个直觉是：MoE通过不同系数加权得到的FF...\nAli Hatamizadeh\n: Thank you for looking into this...\n苏剑林\n: 对于线性层，取max能得到谱范数，然后可以推出Muon，Muo...\n苏剑林\n: Thanks for sharing! I looked in...\nPurshow\n: 苏老师，请问下取max会不会影响控制下界太宽松了呢\nbookworm\n: 天下苦秦久矣，期待苏老师的教材！\nAli Hatamizadeh\n: Hi ! I am the lead GDN-2 author...\nyixiao chen\n: 谢谢苏神！代码在https://github.com/Chyx...\n苏剑林\n: 欢迎作者！我仔细读了一下，感觉写作还是很清晰的，至少比 htt...\n苏剑林\n: 免训练估计够呛，从实用的角度看，其实也不应该相信任何免训练的可能\nUSERLOGIN\n登录\n科学空间|Scientific Spaces\n登录\n打赏\n公式\n天象\n链接\n时光\n博览\n归档\n渴望成为一个小飞侠\n欢迎订阅\n个性邮箱\n天象信息\n观测ISS\nLaTeX\n关于博主\n欢迎访问“科学空间”，这里将与您共同探讨自然科学，回味人生百态；也期待大家的分享～\n千奇百怪\nEverything\n天文探索\nAstronomy\n数学研究\nMathematics\n物理化学\nPhy-chem\n信息时代\nBig-Data\n生物自然\nBiology\n图片摄影\nPhotograph\n问题百科\nQuestions\n生活/情感\nLife-Feeling\n资源共享\nResources\n千奇百怪\n天文探索\n数学研究\n物理化学\n信息时代\n生物自然\n图片摄影\n问题百科\n生活/情感\n资源共享\n首页\n信息时代\nTransformer升级之路：14、当HWFA遇见ReRoPE\n24\nAug\nTransformer升级之路：14、当HWFA遇见ReRoPE\nBy\n苏剑林\n|\n2023-08-24\n|\n63770位读者\n|\n:\n在上一篇文章\n《Transformer升级之路：13、逆用Leaky ReRoPE》\n中，笔者尝试通过在训练阶段逆用Leaky ReRoPE的思路，使得推理阶段的位置编码变为正常的RoPE，从而在达到长度外推的同时解决ReRoPE推理变慢的缺点。遗憾的是，从实验结果来看，“Leaky ReRoPE → RoPE”的效果并不如“RoPE → ReRoPE/Leaky ReRoPE”，因此这个问题尚未完全解决。\n此时，笔者想到此前在\n《Transformer升级之路：9、一种全局长度外推的新思路》\n提出的HWFA本身就具有一定的长度外推能力，如果跟ReRoPE“强强联合”，是否会有更好的效果？更关键是，HWFA的加入可以大幅度降低推理成本，从而弥补ReRoPE的不足！\n温故\n#\n首先，“例行公事”地回顾一下HWFA。HWFA（Hybird Window-Full Attention）并非一个具体的模型，而是一种Attention的组合方式，能够在基本保持效果不变的前提下，增强Attention模型的长度外推能力，同时还能降低训练和推理成本。\n具体来说，HWFA是“$L-1$层Window RoPE Attention + $1$层Full NoPE Attention”，即前面$L-1$层Attention都加上\nRoPE\n，并通过window限制感受野，这样一来推理成本就变为常数，并且基于block parallel进行优化的话，也可以提升训练速度；至于最后一层Attention，则保留global的形式，但去掉位置编码（NoPE），同时加上\n$\\log n$缩放\n。经过这样修改，并且适当选择window之后，模型的训练效果只有轻微下降，同时呈现出优秀的长度外推能力。\n无独有偶，后来Google提出了\nFOT（Focused Transformer）\n，它跟HWFA有很多异曲同工之处：同样是$L-1$层Local Attention加$1$层Full Attention，Full Attention同样是NoPE的，不同的是FOT把Full Attention放在中间，并且Local Attention没有严格限制感受野，所以无法直接长度外推，因此它提出了crossbatch training来拓展模型长度。事后，笔者实验过在HWFA上使用crossbatch training，也有不错的效果。\n知新\n#\n回到本文的主题，HWFA如何跟ReRoPE“强强联合”呢？我们知道，ReRoPE是用在Full RoPE Attention上的，就是在推理阶段截断一下相对位置矩阵：\n$$\\begin{pmatrix}0 & \\\\\n1 & 0 & \\\\\n2 & 1 & 0 &\\\\\n\\ddots & 2 & 1 & 0 & \\\\\n\\ddots & \\ddots & 2 & 1 & 0 & \\\\\n\\ddots & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots \\\\\n\\small{L - 2} & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots & \\ddots \\\\\n\\small{L - 1} & \\small{L - 2} & \\ddots & \\ddots & \\ddots & 2 & 1 & 0 & \\\\\n\\end{pmatrix} \\,\\to\\, \\begin{pmatrix}\n\\color{red}{0} & \\\\\n\\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{\\ddots} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\ddots} & \\color{green}{w} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\color{red}{\\ddots} & \\\\\n\\color{green}{w} & \\color{green}{\\ddots} & \\color{green}{\\ddots} & \\color{green}{w} & \\color{red}{\\small{w - 1}} & \\color{red}{\\ddots} & \\color{red}{1} & \\color{red}{0} & \\\\\n\\end{pmatrix}$$\n出人意料的是，这样的事后处理体现出极佳的长度外推能力。然而，由于RoPE的特殊性，原始的ReRoPE实现需要算两次Attention矩阵，并且不兼容主流的Flash Attention加速等。总的来说，推理阶段的成本增加略有点大。\n不过，HWFA的加入将会极大地缓解这个问题！综上所述，ReRoPE只用在Full RoPE Attention上，HWFA则大部分都是Window RoPE Attention，所以“HWFA+ReRoPE”的方案就呼之欲出了：训练阶段将HWFA原本的Full NoPE Attention换成Full RoPE Attention，然后推理阶段则改为Full ReRoPE Attention。这样一来推理阶段切换ReRoPE带来的额外成本就会变得非常少，而且其他层换为Window Attention带来的收益更加显著。\n除此之外，“HWFA+ReRoPE”还可以弥补原本HWFA的效果损失。此前，为了保证长度外推能力，HWFA的Full Attention要去掉位置编码（即NoPE），同时Window Attention的感受野$\\tilde{w}$要满足$(\\tilde{w}-1)(L-1)+1 = \\alpha N$（其中$L$是层数，$N$是训练长度，$0 < \\alpha \\leq 1$），这些约束限制了模型的表达能力，导致了训练效果变差。而引入ReRoPE之后，Window Attention的感受野可以适当取大一些，Full Attention也可以用RoPE，还可以将它放到中间层而不单是最后一层，甚至也可以多于$1$层Full Attention。这些变化都可以弥补效果损失，并且得益于ReRoPE，长度外推能力并不会有所下降。\n为了区别最初版的HWFA，我们也可以将“HWFA+ReRoPE”的组合，称为“HWFA2”。\n实验\n#\n下面分享一些“HWFA+ReRoPE（HWFA2）”的实验结果。由于引入ReRoPE之后，HWFA的自由度就大了很多，因此下降只是挑笔者认为比较符合直觉的组合进行实验，无法充分验证所有排列组合。\n实验模型跟之前的HWFA、ReRoPE的一样，都是1亿参数的GAU模型，512的训练长度。\n注意这里有两个window参数：一个是ReRoPE本身有个$w$参数，此前ReRoPE实验显示这个影响不大，所以下面统一取256；另一个是HFWA的Window Attention的感受野，上面记为$\\tilde{w}$，这是可调的。\n所以，“HWFA+ReRoPE”的主要参数就是Window Attention的$\\tilde{w}$，以及Full Attention的层数和分布位置。此前笔者做了一些对比实验，显示从训练效果来看，Full Attention放在中间要比放在末尾效果更好，所以如果是1层Full Attention，那么它默认的放置位置是(index = num_layers / 2)的层，如果是2层Full Attention，那么默认的放置位置是(index = num_layers / 3)和(index = 2 * num_layers / 3)的层，依此类推。\n部分实验结果如下：\n\\begin{array}{c|cc}\n\\hline\n\\text{测试长度} & 512(\\text{训练}) & 4096(\\text{重复}) & 4096(\\text{不重复})\\\\\n\\hline\n\\text{Baseline} & 49.41\\% & 24.17\\% & 23.16\\% \\\\\n\\text{Baseline-}\\log n & 49.40\\% & 24.60\\% & 24.02\\% \\\\\n\\hline\n\\text{ReRoPE-w256} & 49.41\\% & 77.90\\% & 48.48\\% \\\\\n\\text{ReRoPE-w256-}\\log n^{\\color{red}{\\dagger}} & 49.41\\% & 82.40\\% & 48.85\\% \\\\\n\\text{ReRoPE-w256-}\\log n & 49.40\\% & \\boldsymbol{85.12\\%} & 49.07\\% \\\\\n\\hline\n\\text{InvLeaky ReRoPE-w128-}\\log n & 49.38\\% & 82.25\\% & 48.32\\% \\\\\n\\text{InvLeaky ReRoPE-w128-b8-}\\log n & 49.62\\% & 81.15\\% & 48.85\\% \\\\\n\\hline\n\\text{HFWA} & 48.70\\% & 80.84\\% & 48.15\\% \\\\\n\\hline\n\\text{HFWA-ReRoPE-w32-f1} & 49.29\\% & 83.13\\% & 49.34\\% \\\\\n\\text{HFWA-ReRoPE-w64-f1} & 49.32\\% & 82.41\\% & \\boldsymbol{49.37\\%} \\\\\n\\text{HFWA-ReRoPE-w128-f1} & 49.21\\% & 80.18\\% & 48.99\\% \\\\\n\\text{HFWA-ReRoPE-w256-f1} & 49.00\\% & 54.94\\% & 47.64\\% \\\\\n\\text{HFWA-ReRoPE-w32-f2} & \\boldsymbol{49.50}\\% & 84.09\\% & 49.35\\% \\\\\n\\text{HFWA-ReRoPE-w64-f2} & 49.46\\% & 84.43\\% & 49.36\\% \\\\\n\\text{HFWA-ReRoPE-w128-f2} & 49.35\\% & 83.09\\% & 48.97\\% \\\\\n\\text{HFWA-ReRoPE-w256-f2} & 49.37\\% & 75.24\\% & 48.42\\% \\\\\n\\hline\n\\end{array}\n上表中$\\text{w}$后的数字就是Window Attention的感受野$\\tilde{w}$的大小，$\\text{f}$后的数字就是Full Attention的层数。原本的HFWA由于各种约束，$\\tilde{w}$只取到了16，再大的话长度外推能力就会明显下降。而从上表可以看到，增大了$\\tilde{w}$后，训练性能可以迅速对齐baseline，并且进一步增加Full Attention还超过了baseline。至于外推效果，$\\text{w32},\\text{w64}$这两个case都相当不错，明显超过了HFWA。总的来看，HFWA-ReRoPE的最佳组合是$\\text{w64-f2}$，训练效果和不重复的外推效果都超过了原本的ReRoPE，再结合训练长度$N$是512、层数$L$是24来看，猜测$\\tilde{w}$的最佳取值应该是$N/L$的$2\\sim 4$倍左右。\n小结\n#\n本文提出了HWFA与ReRoPE的组合使用方式，小规模的实验结果显示，这种组合能够在不损失训练效果的同时，达到近乎最佳的长度外推效果，并且得益于HFWA的设计，还可以明显地降低推理成本，有效地缓解了ReRoPE原本的推理成本增加的缺点。\n转载到请包括本文地址：\nhttps://spaces.ac.cn/archives/9731\n更详细的转载事宜请参考：\n《科学空间FAQ》\n如果您还有什么疑惑或建议，欢迎在下方评论区继续讨论。\n如果您觉得本文还不错，欢迎\n分享\n/\n打赏\n本文。打赏并非要从中获得收益，而是希望知道科学空间获得了多少读者的真心关注。当然，如果你无视它，也不会影响你的阅读。再次表示欢迎和感谢！\n打赏\n微信打赏\n支付宝打赏\n因为网站后台对打赏并无记录，因此欢迎在打赏时候备注留言。\n你还可以\n点击这里\n或在下方评论区留言来告知你的建议或需求。\n如果您需要引用本文，请参考：\n苏剑林. (Aug. 24, 2023). 《Transformer升级之路：14、当HWFA遇见ReRoPE 》[Blog post]. Retrieved from\nhttps://spaces.ac.cn/archives/9731\n@online{kexuefm-9731,\ntitle={Transformer升级之路：14、当HWFA遇见ReRoPE},\nauthor={苏剑林},\nyear={2023},\nmonth={Aug},\nurl={\\url{https://spaces.ac.cn/archives/9731}},\n}\n分类：\n信息时代\n标签：\nattention\n,\n位置编码\n,\n外推\n,\nrope\n22 评论\n<\nTransformer升级之路：13、逆用Leaky ReRoPE\n|\nLion/Tiger优化器训练下的Embedding异常和对策\n>\n你也许还对下面的内容感兴趣\n如何让GDN2更稳定一些？\n简单谈谈K3的MoE和Attention\n将Softmax Attention线性化为Gated DeltaNet\nLogSumExp和Softmax的泰勒展开\nAttention Residuals 回忆录\nDeltaNet的核心逆矩阵的元素总是在[-1, 1]内\n为什么DeltaNet要加L2 Normalize？\n低精度Attention可能存在有偏的舍入误差\n为什么线性注意力要加Short Conv？\nQK-Clip：让Muon在Scaleup之路上更进一步\n发表你的看法\nWinky\nAugust 31st, 2023\n你好，请问这句话该如何理解呢'推理阶段则改为Full ReRoPE Attention。这样一来推理阶段切换ReRoPE带来的额外成本就会变得非常少',不还是需要计算两次attention吗\n回复评论\n苏剑林\n发表于\n August 31st, 2023\n假设模型是24层，原来是24层attention都要计算两次，现在是只有2层attention要计算两次，并且另外22层attention可以明显地减少计算量，一打一消之下，最后还提速了。\n回复评论\nWinky 发表于\n September 1st, 2023\n好的！理解了！总体上考虑\n回复评论\n汐风牧神\nSeptember 19th, 2023\n苏神，我们最近发现一种能够大幅提升RoPE外推性能的新方法，Collinear Constrained Attention，是否有兴趣探讨看看，https://arxiv.org/abs/2309.08646\n回复评论\n苏剑林\n发表于\n September 19th, 2023\n早上在arxiv已经刷到了，谢谢推荐。扫了一眼似乎是个新颖的思路，等我认真读读再来请教。\n回复评论\n汐风牧神 发表于\n September 19th, 2023\n好的，我们的代码开源了，https://github.com/luban-agi/coca，感谢苏神一起探讨\n回复评论\n汐风牧神 发表于\n September 21st, 2023\n由于公司规章制度，github暂时关闭了，近期完成相关审批后会重新开放\n回复评论\n苏剑林\n发表于\n September 23rd, 2023\n已经简单实验过，效果不明显。邮件你没有回复～\n回复评论\n汐风牧神 发表于\n September 25th, 2023\n邮件是发到我在论文填写的一作的公司邮箱吗？我找了一遍没有找到邮件，垃圾箱里也没有找到。苏神可以确认一下邮箱是否有误。邮件里可以简单附一下实验设置，以及基于哪个框架下使用的CoCA，我看下问题\n回复评论\n苏剑林\n发表于\n September 25th, 2023\n是你在这留言所填的QQ邮箱。\n汐风牧神 发表于\n September 25th, 2023\n是不是邮箱名字漏掉了.xxx，邮箱格式是xxxx.xxx@xxx.com\n回复评论\n汐风牧神 发表于\n September 25th, 2023\n苏神，虽然没有看到邮件，但我猜测问题可能出在RoPE旋转的维度与CoCA共线约束的维度没有统一，例如：如果RoPE是旋转（0,1）....（d-2,d-1）的维度，那么CoCA共线约束也需要保持这样的维度分配；而如果RoPE是像LlaMa一样旋转（0,d/2）...（d/2-1,d-1）的维度，那么CoCA共线约束同理。可以确认下是否这个问题导致\n回复评论\n汐风牧神 发表于\n September 25th, 2023\n可以发送到我论文里留的公司邮箱，公司电脑因为不能登陆个人邮箱，很少用\n回复评论\n苏剑林\n发表于\n September 25th, 2023\n已转发过去。\n回复评论\nshomy\nNovember 3rd, 2023\n苏神 有个小白问题，请教一下，关于现在用RoPE的大模型长序列或者外推建模，假设到32k，两种方式哪种会更好些呢：\n（1）直接在预训练阶段拉到32k，不依赖其他措施。\n（2）预训练阶段比较短，比如到4k或者8k，后面通过其他外推或插值的方式比如YaRN等，到32k。\n多谢苏神\n回复评论\n苏剑林\n发表于\n November 10th, 2023\n假如训练总token数一样，单纯比效果的话，肯定是第一种方案更好啊。\n回复评论\nlubu\nNovember 7th, 2023\n苏神您好！我想复现并测试一下HWFA+ReRope，但一直没有成功，请问可以要一份源码到邮箱吗？\n回复评论\n苏剑林\n发表于\n November 10th, 2023\n发了一份到你这个邮箱。但代码是tf的，估计不好用，建议仅供参考。\n回复评论\nlodestar\nNovember 20th, 2024\n苏神您好，我想复现一下这篇博客的结果，请问可以发一份源码到我邮箱吗？\n回复评论\n苏剑林\n发表于\n November 24th, 2024\n有点难，代码倒是可以发你一份，但数据没法发，要对齐还是有点困难的。还有主要这是我早期一直跑下来的代码，还是tf+bert4keras的，现在已经不大好意思见人了。\n回复评论\nlodestar 发表于\n November 27th, 2024\n哦这样，那就把代码发我吧，就算跑不起来也能参考一下。\n回复评论\n苏剑林\n发表于\n December 1st, 2024\n已发\n回复评论\n取消回复\n你的大名\n电子邮箱\n个人网站（选填）\n1. 可以使用LaTeX代码，点击“预览效果”可查看效果；\n2. 可以通过点击评论楼层编号来引用该楼层；\n3. 网站可能会有点卡，如非确认评论失败，请\n不要重复点击提交\n。\n内容速览\n温故\n知新\n实验\n小结\n智能搜索\n支持整句搜索！网站自动使用\n结巴分词\n进行分词，并结合ngrams排序算法给出合理的搜索结果。\n热门标签\nattention\n生成模型\n优化\n模型\n语言模型\n梯度\n优化器\n矩阵\n概率\n网站\n转载\n微分方程\n分析\n天象\n深度学习\n积分\npython\n几何\n扩散\n力学\n无监督\n损失函数\n节日\n线性\n生活\n随机文章\n行动起来！共同应对全球气候变暖\n最小熵原理（三）：“飞象过河”之句模版和语言结构\n几何的数与数的几何：超复数的浅探究\n【致敬】费曼诞辰100年\n【奥赛之行】非同一般的天文奥赛\n圆周率节快乐！|| 原来已经写了十年博客～\n科学空间“微信群|聊天机器人”上线测试\n高斯型积分的微扰展开（一）\n生成扩散模型漫谈（二十七）：将步长作为条件输入\n细水长flow之NICE：流模型的基本概念与实现\n最近评论\ncarl_lei\n: 苏神，之前我一直有一个直觉是：MoE通过不同系数加权得到的FFN相比原始的大FFN（也就是直接...\nAli Hatamizadeh\n: Thank you for looking into this and featuring i...\n苏剑林\n: 对于线性层，取max能得到谱范数，然后可以推出Muon，Muon比Adam、SGD等要好，所以...\n苏剑林\n: Thanks for sharing! I looked into this improvem...\nPurshow\n: 苏老师，请问下取max会不会影响控制下界太宽松了呢\nbookworm\n: 天下苦秦久矣，期待苏老师的教材！\nAli Hatamizadeh\n: Hi ! I am the lead GDN-2 author.We've addressed...\nyixiao chen\n: 谢谢苏神！代码在https://github.com/Chyxx/CyclicFlowAtte...\n苏剑林\n: 欢迎作者！我仔细读了一下，感觉写作还是很清晰的，至少比 https://papers.cool...\n苏剑林\n: 免训练估计够呛，从实用的角度看，其实也不应该相信任何免训练的可能\n友情链接\nCool Papers\n数学研发\nSeatop\nXiaoxia\n积分表-网络版\n丝路博傲\n数学之家\n有趣天文奇观\nTwistedW\ngodweiyang\nAI柠檬\n王登科-DK博客\nESON\n枫之羽\ncoding-zuo\n博科园\n孔皮皮的博客\n运鹏的博客\njiming.site\nOmegaXYZ\nEAI猩球\n文举的博客\n汇尘轩\n申请链接\n本站采用创作共用版权协议，要求署名、非商业用途和保持一致。转载本站内容必须也遵循“\n署名-非商业用途-保持一致\n”的创作共用协议。\n© 2009-2026 Scientific Spaces. All rights reserved. Theme by\nlaogui\n. Powered by\nTypecho\n. 备案号:\n粤ICP备09093259号-1/2\n。", "metadata": { "title": "Transformer升级之路：14、当HWFA遇见ReRoPE - 科学空间|Scientific Spaces", "description": "在上一篇文章《Transformer升级之路：13、逆用Leaky ReRoPE》中，笔者尝试通过在训练阶段逆用Leaky ReRoPE的思路，使得推理阶段的位置编码变为正常的RoPE，从而在达...", "keywords": "attention,位置编码,外推,rope", "author": null }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/images/slide-button.png", "bytes": 4328, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-fcd8b7d1212b1f5eab2b.png", "src": "https://spaces.ac.cn/usr/themes/geekg/images/slide-button.png", "alt": "MobileSideBar", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/images/rss.png", "bytes": 19283, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-043e6308dc873ff9fa67.png", "src": "https://spaces.ac.cn/usr/themes/geekg/images/rss.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/images/mail.png", "bytes": 16811, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-942ff5c7b650f4d3d31f.png", "src": "https://spaces.ac.cn/usr/themes/geekg/images/mail.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/images/Saturn.png", "bytes": 2524, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-53b9d00850633d612de3.png", "src": "https://spaces.ac.cn/usr/themes/geekg/images/Saturn.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/images/iss.png", "bytes": 3301, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-896b482bdde5ec1df88a.png", "src": "https://spaces.ac.cn/usr/themes/geekg/images/iss.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/images/pi.png", "bytes": 939, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-3f7f2f2318249e4eac9a.png", "src": "https://spaces.ac.cn/usr/themes/geekg/images/pi.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/images/mlogo.png", "bytes": 18685, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-cbbace78f2cd4551d156.png", "src": "https://spaces.ac.cn/usr/themes/geekg/images/mlogo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/payment/wx.png", "bytes": 36540, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-c5d79f289db1f5d431bc.png", "src": "https://spaces.ac.cn/usr/themes/geekg/payment/wx.png", "alt": "科学空间", "downloaded": true }, { "status": "ok", "url": "https://spaces.ac.cn/usr/themes/geekg/payment/zfb.png", "bytes": 31565, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-4345e58697a529a3d3a5.png", "src": "https://spaces.ac.cn/usr/themes/geekg/payment/zfb.png", "alt": "科学空间", "downloaded": true }, { "status": "ok", "url": "https://kexue.fm/usr/themes/geekg/images/cc.gif", "bytes": 2031, "reused": false, "local_path": "attachments/f74c12ee5ad8/images/body-b2904ced8e4508d61bed.gif", "src": "https://kexue.fm/usr/themes/geekg/images/cc.gif", "alt": "署名-非商业用途-保持一致", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://spaces.ac.cn/reward.html", "label": "打赏", "type": "a" }, { "href": "https://spaces.ac.cn/latex.html", "label": "公式", "type": "a" }, { "href": "https://spaces.ac.cn/ac.html", "label": "天象", "type": "a" }, { "href": "https://spaces.ac.cn/links.html", "label": "链接", "type": "a" }, { "href": "https://spaces.ac.cn/me.html", "label": "时光", "type": "a" }, { "href": "https://spaces.ac.cn/science.html", "label": "博览", "type": "a" }, { "href": "https://spaces.ac.cn/content.html", "label": "归档", "type": "a" }, { "href": "https://spaces.ac.cn/category/Everything", "label": "千奇百怪", "type": "a" }, { "href": "https://spaces.ac.cn/category/Astronomy", "label": "天文探索", "type": "a" }, { "href": "https://spaces.ac.cn/category/Mathematics", "label": "数学研究", "type": "a" }, { "href": "https://spaces.ac.cn/category/Phy-chem", "label": "物理化学", "type": "a" }, { "href": "https://spaces.ac.cn/category/Big-Data", "label": "信息时代", "type": "a" }, { "href": "https://spaces.ac.cn/category/Biology", "label": "生物自然", "type": "a" }, { "href": "https://spaces.ac.cn/category/Photograph", "label": "图片摄影", "type": "a" }, { "href": "https://spaces.ac.cn/category/Questions", "label": "问题百科", "type": "a" }, { "href": "https://spaces.ac.cn/category/Life-Feeling", "label": "生活/情感", "type": "a" }, { "href": "https://spaces.ac.cn/category/Resources", "label": "资源共享", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11945", "label": "如何让GDN2更稳定一些？", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11928", "label": "通过微扰分析求解等式约束优化问题", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11917", "label": "慢即是快：1. Shampoo的尽...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11910", "label": "排序不等式及其推广", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11885", "label": "让炼丹更科学一些（十）：单调性假设...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11882", "label": "让炼丹更科学一些（九）：经典自适应...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11879", "label": "让炼丹更科学一些（八）：多阶段训练...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11875", "label": "动量的新理解：逼近特征层面的梯度下降", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11864", "label": "流形上的最速下降：7. Stief...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11854", "label": "除了交叉熵，LM Loss还有什么选择？", "type": "a" }, { "href": "https://spaces.ac.cn/archives/10699/comment-page-3", "label": "carl_lei : 苏神，之前我一直有一个直觉是：MoE通过不同系数加权得到的FF...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11945/comment-page-1", "label": "Ali Hatamizadeh : Thank you for looking into this...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11340/comment-page-1", "label": "苏剑林 : 对于线性层，取max能得到谱范数，然后可以推出Muon，Muo...", "type": "a" }, { "href": "https://spaces.ac.cn/archives/1328/comment-page-1", "label": "bookworm : 天下苦秦久矣，期待苏老师的教材！", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11848/comment-page-2", "label": "苏剑林 : 免训练估计够呛，从实用的角度看，其实也不应该相信任何免训练的可能", "type": "a" }, { "href": "https://spaces.ac.cn/admin/login.php", "label": "登录", "type": "a" }, { "href": "https://spaces.ac.cn/", "label": "科学空间|Scientific Spaces", "type": "a" }, { "href": "https://spaces.ac.cn/feed", "label": "欢迎订阅", "type": "a" }, { "href": "https://spaces.ac.cn/archives/119", "label": "个性邮箱", "type": "a" }, { "href": "https://spaces.ac.cn/archives/41", "label": "观测ISS", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731", "label": "Transformer升级之路：14、当HWFA遇见ReRoPE", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9728", "label": "《Transformer升级之路：13、逆用Leaky ReRoPE》", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9603", "label": "《Transformer升级之路：9、一种全局长度外推的新思路》", "type": "a" }, { "href": "https://spaces.ac.cn/archives/8265", "label": "RoPE", "type": "a" }, { "href": "https://spaces.ac.cn/archives/8823", "label": "$\\log n$缩放", "type": "a" }, { "href": "https://papers.cool/arxiv/2307.03170", "label": "FOT（Focused Transformer）", "type": "a" }, { "href": "https://spaces.ac.cn/archives/6508", "label": "《科学空间FAQ》", "type": "a" }, { "href": "http://mail.qq.com/cgi-bin/qm_share?t=qm_mailme&email=tN7d1drY3drrx8H0xcWa19vZ", "label": "点击这里", "type": "a" }, { "href": "https://spaces.ac.cn/tag/attention/", "label": "attention", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81/", "label": "位置编码", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E5%A4%96%E6%8E%A8/", "label": "外推", "type": "a" }, { "href": "https://spaces.ac.cn/tag/rope/", "label": "rope", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9736", "label": "Lion/Tiger优化器训练下的Embedding异常和对策", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11848", "label": "简单谈谈K3的MoE和Attention", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11823", "label": "将Softmax Attention线性化为Gated DeltaNet", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11814", "label": "LogSumExp和Softmax的泰勒展开", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11664", "label": "Attention Residuals 回忆录", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11563", "label": "DeltaNet的核心逆矩阵的元素总是在[-1, 1]内", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11486", "label": "为什么DeltaNet要加L2 Normalize？", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11371", "label": "低精度Attention可能存在有偏的舍入误差", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11320", "label": "为什么线性注意力要加Short Conv？", "type": "a" }, { "href": "https://spaces.ac.cn/archives/11126", "label": "QK-Clip：让Muon在Scaleup之路上更进一步", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22613", "label": "回复评论", "type": "a" }, { "href": "https://kexue.fm/", "label": "苏剑林", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22624", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22626", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22741", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22743", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22744", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22765", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22780", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22792", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22793", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22794", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22796", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22798", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=22996", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=23039", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=23008", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=23046", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=25822", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=25840", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=25847", "label": "回复评论", "type": "a" }, { "href": "https://spaces.ac.cn/archives/9731/comment-page-1?replyTo=25879", "label": "回复评论", "type": "a" }, { "href": "https://github.com/fxsjy/jieba", "label": "结巴分词", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E7%94%9F%E6%88%90%E6%A8%A1%E5%9E%8B/", "label": "生成模型", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E4%BC%98%E5%8C%96/", "label": "优化", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E6%A8%A1%E5%9E%8B/", "label": "模型", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E8%AF%AD%E8%A8%80%E6%A8%A1%E5%9E%8B/", "label": "语言模型", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E6%A2%AF%E5%BA%A6/", "label": "梯度", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E4%BC%98%E5%8C%96%E5%99%A8/", "label": "优化器", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E7%9F%A9%E9%98%B5/", "label": "矩阵", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E6%A6%82%E7%8E%87/", "label": "概率", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E7%BD%91%E7%AB%99/", "label": "网站", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E8%BD%AC%E8%BD%BD/", "label": "转载", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E5%BE%AE%E5%88%86%E6%96%B9%E7%A8%8B/", "label": "微分方程", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E5%88%86%E6%9E%90/", "label": "分析", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E5%A4%A9%E8%B1%A1/", "label": "天象", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0/", "label": "深度学习", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E7%A7%AF%E5%88%86/", "label": "积分", "type": "a" }, { "href": "https://spaces.ac.cn/tag/python/", "label": "python", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E5%87%A0%E4%BD%95/", "label": "几何", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E6%89%A9%E6%95%A3/", "label": "扩散", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E5%8A%9B%E5%AD%A6/", "label": "力学", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E6%97%A0%E7%9B%91%E7%9D%A3/", "label": "无监督", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E6%8D%9F%E5%A4%B1%E5%87%BD%E6%95%B0/", "label": "损失函数", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E8%8A%82%E6%97%A5/", "label": "节日", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E7%BA%BF%E6%80%A7/", "label": "线性", "type": "a" }, { "href": "https://spaces.ac.cn/tag/%E7%94%9F%E6%B4%BB/", "label": "生活", "type": "a" }, { "href": "https://spaces.ac.cn/archives/107", "label": "行动起来！共同应对全球气候变暖", "type": "a" }, { "href": "https://spaces.ac.cn/archives/5577", "label": "最小熵原理（三）：“飞象过河”之句模版和语言结构", "type": "a" }, { "href": "https://spaces.ac.cn/archives/2291", "label": "几何的数与数的几何：超复数的浅探究", "type": "a" }, { "href": "https://spaces.ac.cn/archives/5533", "label": "【致敬】费曼诞辰100年", "type": "a" }, { "href": "https://spaces.ac.cn/archives/613", "label": "【奥赛之行】非同一般的天文奥赛", "type": "a" }, { "href": "https://spaces.ac.cn/archives/6469", "label": "圆周率节快乐！|| 原来已经写了十年博客～", "type": "a" }, { "href": "https://spaces.ac.cn/archives/4096", "label": "科学空间“微信群|聊天机器人”上线测试", "type": "a" }, { "href": "https://spaces.ac.cn/archives/3217", "label": "高斯型积分的微扰展开（一）", "type": "a" }, { "href": "https://spaces.ac.cn/archives/10617", "label": "生成扩散模型漫谈（二十七）：将步长作为条件输入", "type": "a" }, { "href": "https://spaces.ac.cn/archives/5776", "label": "细水长flow之NICE：流模型的基本概念与实现", "type": "a" }, { "href": "https://papers.cool/", "label": "Cool Papers", "type": "a" }, { "href": "https://bbs.emath.ac.cn/", "label": "数学研发", "type": "a" }, { "href": "http://www.seatop.com.cn/", "label": "Seatop", "type": "a" }, { "href": "https://xiaoxia.org/", "label": "Xiaoxia", "type": "a" }, { "href": "https://kexue.fm/sci/integral/index.html", "label": "积分表-网络版", "type": "a" }, { "href": "http://blog.dvxj.com/", "label": "丝路博傲", "type": "a" }, { "href": "http://www.2math.cn/", "label": "数学之家", "type": "a" }, { "href": "http://interesting-sky.china-vo.org/", "label": "有趣天文奇观", "type": "a" }, { "href": "http://www.twistedwg.com/", "label": "TwistedW", "type": "a" }, { "href": "https://godweiyang.com/", "label": "godweiyang", "type": "a" }, { "href": "https://blog.ailemon.net/", "label": "AI柠檬", "type": "a" }, { "href": "https://greatdk.com/", "label": "王登科-DK博客", "type": "a" }, { "href": "https://blog.eson.org/", "label": "ESON", "type": "a" }, { "href": "https://fzhiy.net/", "label": "枫之羽", "type": "a" }, { "href": "https://coding-zuo.github.io/", "label": "coding-zuo", "type": "a" }, { "href": "https://www.bokeyuan.net/", "label": "博科园", "type": "a" }, { "href": "https://www.kppkkp.top/", "label": "孔皮皮的博客", "type": "a" }, { "href": "https://yunpengtai.top/", "label": "运鹏的博客", "type": "a" }, { "href": "https://jiming.site/", "label": "jiming.site", "type": "a" }, { "href": "https://www.omegaxyz.com/", "label": "OmegaXYZ", "type": "a" }, { "href": "https://www.robotech.ink/", "label": "EAI猩球", "type": "a" }, { "href": "https://liwenju0.com/", "label": "文举的博客", "type": "a" }, { "href": "https://kirigaya.cn/", "label": "汇尘轩", "type": "a" }, { "href": "https://kexue.fm/links.html", "label": "申请链接", "type": "a" }, { "href": "http://creativecommons.org/licenses/by-nc-nd/2.5/cn/", "label": "", "type": "a" }, { "href": "http://www.laogui.com/", "label": "laogui", "type": "a" }, { "href": "http://typecho.org/", "label": "Typecho", "type": "a" }, { "href": "https://beian.miit.gov.cn/", "label": "粤ICP备09093259号-1/2", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:53:07Z", "source_url": "https://spaces.ac.cn/archives/9731", "papers": [], "actual_video_files": [], "body_chars": 11010, "expected_body_images": 10, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/f74c12ee5ad8/pages/source.html", "local_html": "attachments/f74c12ee5ad8/pages/article.html", "body_base_path": "attachments/f74c12ee5ad8/pages/", "body_markdown_path": "attachments/f74c12ee5ad8/pages/article.md" }