Daily Papers · 2026-10-02
📄 论文日报 | 来源：Hugging Face Daily Papers
🗓 Episode：2026.10.01 | RIDE外推教师残差；UniEvo-VL在线自蒸馏多模态
1. 教师是方向，而非终点：在在线策略蒸馏中外推强化学习诱导的表征残差
- 英文标题：The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
- arXiv：2609.36484
💡 一句话解读：在线蒸馏中学生常被老师“带偏”：输出层把老师内部的变化压扁，还放大噪声，训练不稳。RIDE 改在内部表示层直接外推老师相对其强化学习前版本的位移，让学生沿这个方向走得更远，从而稳定地接近甚至超过老师。
在线策略蒸馏（OPD）训练学生在自身轨迹上匹配教师的下一词元分布，并已取得显著经验收益。其泛化变体允许学生通过在输出空间外推隐式奖励来超越教师。然而，语言模型头会各向异性地衰减这种变化：教师隐藏状态中编码的许多变化仅以很小比例传递到 logits，而输出空间外推所依赖的采样词元对数概率比会引入噪声，外推会放大该噪声，导致训练不稳定。我们观察到强化学习（RL）会使模型内部表示相对于其基础检查点发生偏移，且该偏移方向可在每一层测量。受此观察启发，我们提出 RIDE（RL 诱导方向外推），直接在表示空间中外推 RL 诱导的变化：在每一层和每个词元位置，RIDE 计算教师与其 RL 前检查点之间的残差，并将学生的隐藏状态回归到沿该残差方向超出教师的目标。以采样轨迹为条件，该回归等价于在以教师为中心的二次惩罚下最大化由残差定义的线性方向奖励，这明确了目标如何沿 RL 诱导方向移动学生，同时限制其偏离教师。在涵盖不同规模、架构和预训练谱系的四对基础/RL 教师组合上，RIDE 在每一对上都接近或超过 RL 训练的教师，并且是唯一在均值上做到这一点的方法，同时它始终优于输出空间外推——后者在教师接近其基础模型时会损害学生。项目页面：https://github.com/xixixixixxxx/RIDE。
📄 English Abstract
On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The language-model head, however, attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at a small fraction of its weight, and the sampled-token log-probability ratios on which output-space extrapolation relies inject noise that the extrapolation amplifies, making training unstable. We observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Motivated by this observation, we propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space: at every layer and token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual. Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher, which makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher. Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so, and it consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. Project page: https://github.com/xixixixixxxx/RIDE.
2. UniEvo-VL：面向多模态模型自我改进的在线策略自蒸馏训练方案
- 英文标题：UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement
- arXiv：2609.38721
💡 一句话解读：这篇论文想解决多模态模型在测试阶段无法利用自身反馈自我提升的问题。方法是让同一个模型扮演老师和学生：学生只看原题，老师额外看到模型自己写的批评意见，然后让两者的生成分布尽量一致，从而把自我纠错能力内化。
现代多模态模型将生成与理解整合进一个统一系统，使其能够提供并学习自身反馈。受这种统一能力启发，我们提出 UniEvo-VL，一个面向多模态模型的自进化框架，使其在测试时计算中从这种建设性的自我纠错反馈中学习。我们不依赖单独的、通常更大的教师模型，而是将模型的自我批评作为特权信息，让单个多模态模型在不同上下文下同时充当教师和学生。学生只看到原始问题，而教师则以特权批评为条件；训练目标是最小化二者在学生自身采样轨迹上逐状态去噪扩散分布之间的差异。实验表明，UniEvo-VL 提升了多模态模型的图像生成能力，同时保持其对额外反思信息的敏感性。具体而言，我们在开源 Qwen-image-2512 基础上构建，观察到 GenEval 从 0.747 显著提升至 0.808，GenEval2 Soft-TIFA 从 32.97 提升至 35.53。此外，使用更强大的外部批评者（如 GPT5.6-Luna）的尝试表明，具备强评判能力的多模态模型可以预期更高的自进化上限。最后，混合文本渲染结果表明，我们的自我改进在不同任务上可能并不均匀。本研究旨在为当前热门的递归自我改进研究方向提供启示，以在无外部监督或指导的情况下提升用户使用多模态模型的体验。
📄 English Abstract
Modern multimodal models bring generation and understanding into a single unified system, which enables them to provide and learn from their own feedback. Motivated by this unified capacity, we introduce UniEvo-VL, a self-evolving framework for multimodal models to learn from this constructive self-correction feedback during test-time compute. Instead of relying on a separate, often larger, teacher, we leverage their self-critiques as privileged information and ask a single multimodal model to act as both teacher and student with different contexts. The student only sees the vanilla question, while the teacher conditions on the privileged critique. Then training minimizes the per-state divergence between their denoising diffusion distributions over the student's own sampling trajectories. Experiments demonstrate that UniEvo-VL improves the image generation capabilities of multimodal models, while maintaining their sensitivity to additional reflection information. Specifically, we build on top of the open-source Qwen-image-2512 and observe a significant performance gain from 0.747 to 0.808 on GenEval and from 32.97 to 35.53 on GenEval2 Soft-TIFA. Moreover, attempts with more powerful external critics (e.g., GPT5.6-Luna) show that multimodal models with strong judge capabilities can anticipate a higher self-evolving ceiling. Last but not least, mixed text-rendering outcomes show that our self-improvements may not be uniform across different tasks. Our study aims to shed light on the current hot recursive self-improvement research line to enhance the user experience when using multimodal models without external supervision or guidance.
3. 虚假前沿：诊断与缓解自演化搜索智能体中的共同作弊
- 英文标题：False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents
- arXiv：2609.39102
💡 一句话解读：自演化搜索智能体在自我训练时，出题者和答题者会一起“作弊”：它们互相认同错误答案，内部奖励涨了，但实际正确率没提高。论文先提出多采样验证来缓解，但效果有限，于是提出CrossFit方法，把出题用的资料分成两组，用另一组训练出的辅助答题者来打分，从而切断作弊反馈，大幅减少虚假一致并提升下游搜索性能。
自演化搜索智能体通过联合优化生成问题的提议者（proposer）和回答问题的求解者（solver）来构建自己的训练课程。这种闭环引入了一种我们称为“共同作弊”（co-cheating）的失效模式：提议者和求解者越来越认同共同的错误，导致内部奖励提升而外部正确性没有相应增益。基于源证据的事后审计显示，随着自演化轮次增加，共同作弊愈发严重，伪标签正确率停滞或下降，即使环内训练信号在改善。最直接的缓解方法是在训练前验证提议：我们引入多采样验证（MSV），它用源文档查询同一模型三次、不用源文档查询三次，以决定任务是否接纳并替换不可靠的伪标签。MSV部分减少了虚假一致，但仍留下大量残余共同作弊，且每个候选需要额外六次标注器生成。这些局限促使我们提出主要方法CrossFit：它将提议者的源文档划分为A组和B组；由A生成的问题由仅在B上训练的辅助求解者评分，反之亦然。交叉拟合的一致性决定提议者奖励，因此同源伪标签无法通过反馈求解者重现，而原始求解者的更新规则不变。用Qwen3.5-4B和Qwen3.5-9B重新运行循环，MSV将虚假一致质量从6.1%降至5.7%、从8.8%降至7.2%，而CrossFit将其降至3.0%和3.7%。用排除源文档的反馈重放相同提议，进一步将虚假一致降至0.4%和0.1%，从而将反馈来源与课程变化隔离开来。在七个下游搜索基准上，CrossFit在4B和9B模型上分别比标准耦合自演化平均性能提升8.8和8.4分，比Search-R1提升8.7和7.8分。
📄 English Abstract
Self-evolving search agents build their own training curricula by jointly optimizing a proposer that generates questions and a solver that answers them. This closed loop introduces a failure mode we call co-cheating: the proposer and solver increasingly agree on shared errors, so internal reward improves without a matching gain in external correctness. A post-hoc audit against source evidence shows co-cheating growing more severe over successive rounds of self-evolution, with pseudo-label correctness stagnating or declining even as the in-loop training signal improves. The most direct mitigation is to verify proposals before training: we introduce multi-sample verification (MSV), which queries the same model three times with the source and three times without it to decide task admission and replace unreliable pseudo-labels. MSV partially reduces false agreement but leaves substantial residual co-cheating and costs six extra labeler generations per candidate. These limitations motivate CrossFit, our main method: it partitions the proposer's source documents into groups A and B; questions generated from A are scored by an auxiliary solver trained only on B, and vice versa. The cross-fitted agreement determines proposer reward, so a same-source pseudo-label cannot be reproduced through the feedback solver, while the original solver's update rule is unchanged. Rerunning the loop with Qwen3.5-4B and Qwen3.5-9B, MSV reduces false-agreement mass from 6.1% to 5.7% and from 8.8% to 7.2%, whereas CrossFit reduces it to 3.0% and 3.7%. Replaying identical proposals with source-excluded feedback further reduces false agreement to 0.4% and 0.1%, isolating feedback ancestry from curriculum changes. Across seven downstream search benchmarks, CrossFit improves average performance over standard coupled self-evolution by 8.8 and 8.4 points and over Search-R1 by 8.7 and 7.8 points at 4B and 9B.
4. AREX-2：通过长时程反思任务推进自我改进智能体
- 英文标题：AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks
- arXiv：2609.38288
💡 一句话解读：这篇研究想让 AI 智能体像人一样反复打磨自己的答案，越改越好。难点在于既要会反思（想出更好的方案），又要能长时间坚持多轮改进。作者用机器学习编程和算法题生成大量“反复改进”的训练数据，让模型学会这种通用本领，结果在多个测试上表现优异，且给的改进轮数越多成绩越好。
我们提出 AREX-2，旨在推进大语言模型（LLM）智能体的自我改进能力，我们将其定义为在测试时迭代优化解决方案的能力。这种能力依赖两种互补能力：反思（reflection），即产出比当前更好的解决方案；以及长时程执行（long-horizon execution），即在多轮迭代中保持改进的有效性。我们假设这两种能力都与领域无关，因此可以在适合监督的场景中学习。据此，我们从机器学习与算法编程任务中合成了长时程改进轨迹，这两个领域能提供可验证的反馈并奖励持续迭代。基于 Qwen3.8-27B 构建的智能体在该数据上训练后，在 MLE-bench Lite（81.8）和 Frontier-CS（70.7）上取得强劲结果，并迁移到深度研究任务，在 BrowseComp 上达 84.0、HLE 上达 52.6、GAIA 上达 92.2、DeepSearchQA 上达 93.8，且随着迭代轮数预算增加而持续提升。这些结果表明，长时程反思数据是通向自我改进智能体的有效路径。
📄 English Abstract
We present AREX-2, an effort to advance the self-improving capability of LLM agents, which we define as the ability to iteratively refine a solution at test time. This ability rests on two complementary capabilities: reflection, which produces a solution better than the current one, and long-horizon execution, which keeps the iteration effective over many rounds. We hypothesize that both capabilities are domain-agnostic, and can therefore be learned in scenarios that are well suited for supervision. Accordingly, we synthesize long-horizon improvement trajectories from machine learning and algorithmic programming tasks, two domains that offer verifiable feedback and reward sustained iteration. Trained on this data, our agent, built on Qwen3.8-27B, achieves strong results on MLE-bench Lite (81.8) and Frontier-CS (70.7), transfers to deep research with 84.0 on BrowseComp, 52.6 on HLE, 92.2 on GAIA, and 93.8 on DeepSearchQA, and keeps improving as its budget of rounds grows. These results show that long-horizon reflective data is an effective route toward self-improving agents.
5. Mid-Harness：在模型与执行框架之间扩展终端智能体的动作
- 英文标题：Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents
- arXiv：2609.39982
💡 一句话解读：终端智能体靠模型随机生成命令来操作电脑，但生成得出好命令不等于能可靠执行，一条错误命令可能把环境搞坏、拖累后续步骤。本文提出 Mid-Harness：在模型和执行框架之间插入一个“候选动作采样+验证”环节，先多生成几个备选命令并挑出最靠谱的一个再执行，从而提升任务成功率。
终端智能体通过随机的模型生成来采取行动，但能够生成有用的动作并不能保证其可靠执行。一条糟糕的命令（例如错误的软件包安装）可能改变环境，从而阻碍后续进展，即便模型本可以生成更好的替代方案。我们研究在模型与执行框架（model-harness）边界分配测试时计算（test-time compute）能否提升动作可靠性与轨迹成功率，以及这种分配为何有效。为研究这些问题，我们提出 Mid-Harness，它在转发一个动作执行之前对候选动作进行采样与验证，同时保持生成器与执行框架不变。在使用 TMAX-9B 生成器时，弱验证下增加动作采样收效甚微，而一个能力强的验证器则能从同一生成器中挖掘出有用的替代方案。在 TerminalBench-Lite 上，GPT-5.6 Sol 验证器将 Pass@1 从基础智能体的 50.00% 提升至采样 8 个动作时的 68.03%。当同一 TMAX-9B 模型充当验证器时，成对验证（pairwise verification）在所评估的验证机制中表现最佳。将更强验证器的响应蒸馏到 TMAX-9B 中可进一步提升 Pass@1，同时保持动作生成器不变。在 TerminalBench-Lite 上使用 TMAX-9B 时，结合动作扩展与轨迹扩展，相比仅生成更多轨迹，能在更低的估计 token 成本下达到更高的成功率。Mid-Harness 在更多模型、基准和执行框架上也提升了性能。这些发现表明，动作扩展（action scaling）是终端智能体中测试时计算扩展的一个有前景的方向。
📄 English Abstract
Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.
6. EvoDuet：面向科学发现的网络搜索与任务求解双层协同演化
- 英文标题：EvoDuet: Bilevel Co-Evolution of Web Searching and Task Solving for Scientific Discovery
- arXiv：2609.40340
💡 一句话解读：大模型做科学发现式的进化搜索时，常因缺少外部知识而卡住；直接加个网页搜索工具，又会反复搜到同样的页面。EvoDuet 让“答案”和“搜索词”互相带动、一起进化：每轮先判断模型是否缺知识，再决定搜新资料、用旧资料还是不搜，从而更高效地找到更优解。
当进展需要模型所缺乏的外部知识时，基于大语言模型（LLM）的进化搜索会陷入停滞。提供相关文档虽有帮助，但仅仅加入一个网络搜索工具，会随着解的不断变化而反复返回相同的页面。我们提出 EvoDuet，一种在模型参数固定条件下让解与搜索查询协同演化的双层优化方法。每次迭代中，一个检索门控让 LLM 评估自身知识缺口，并选择检索新文档、复用已存文档或不使用文档。内层循环优化查询，并依据文档被预测能带来的解得分对文档排序；外层循环基于这些文档并行生成候选解，并记录已评估的结果以供后续搜索使用。在每轮仅生成一个候选解的 21 项优化任务上，EvoDuet 将 OpenEvolve 的归一化发现增益在 GPT-5.6-Luna 上从 74.1% 提升到 78.0%，在 Gemini-3.8-Flash 上从 61.3% 提升到 82.3%，而 Qwen3.5-9B 未从中获益。我们最佳运行结果在八项任务上超过了此前报告的最佳成绩，包括 Q20 和 Rosetta 上的 Swap Reduction，并在另外三项任务上与之持平。EvoDuet 在 Sums/Diffs 和 Denoising 上也能配合其他脚手架（如 Top-K、EvoX）取得提升，表明其可适用于多种进化搜索脚手架。
📄 English Abstract
Evolutionary search with large language models (LLMs) can stall when progress requires external knowledge the model lacks. Supplying relevant documents helps, but simply adding web search tool can keep returning the same pages as solutions change. We introduce EvoDuet, a bi-level optimization method that co-evolves solutions and search queries with fixed model parameters. At each iteration, a retrieval gate lets the LLM assess its knowledge gap and choose to retrieve new documents, reuse stored ones, or proceed without them. An inner loop refines queries and ranks documents by the solution scores they are predicted to yield; an outer loop generates candidates in parallel from these documents and records the evaluated outcomes for later searches. Across 21 optimization tasks with one candidate per iteration, EvoDuet raises OpenEvolve's normalized discovery gain from 74.1% to 78.0% with GPT-5.6-Luna and from 61.3% to 82.3% with Gemini-3.8-Flash, whereas Qwen3.5-9B does not benefit. Our best runs surpass the previously reported best scores on eight tasks, including Swap Reduction on Q20 and Rosetta, and match them on three more. EvoDuet also improves with other scaffolds (e.g., Top-K, EvoX) on Sums/Diffs and Denoising, demonstrating its applicability across evolutionary search scaffolds.
7. WorldAuditBench：使用多模态智能体进行交互式3D世界审计
- 英文标题：WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents
- arXiv：2609.40325
💡 一句话解读：这篇研究要解决的是：在交互式3D虚拟世界里，如何自动找出不合理的地方，比如悬空的物体、能穿过的墙或与周围场景不搭的东西。作者提出了一个叫WorldAuditBench的测试基准，包含213个异常任务，并比较了两种让AI智能体边探索边判断的方法，结果发现AI的表现远不如人类。
随着交互式3D世界越来越多地被用于研究智能行为，开发高效的流程来识别这些模拟环境中的异常（如悬浮物体、可穿越的墙壁或与周围场景不一致的物体）变得十分重要。多模态AI系统，包括视觉语言模型（VLMs）和视觉-语言-动作模型（VLAs），已显示出自动化此任务的潜力。然而，3D世界审计非常复杂，需要两种不同能力的紧密耦合：动作（在3D世界中导航并系统高效地搜索异常）和视觉推理（理解环境并从多模态观察中识别异常）。多模态智能体能否有效耦合这两种能力——利用视觉推理识别潜在异常，同时采取行动验证它们——在很大程度上仍未被探索。本文介绍了WorldAuditBench，一个用于3D世界审计的基准，包含13个环境中的213个异常任务，这些环境由Unreal Engine 5和Three.js构建，涵盖五个异常类别。我们在固定探索预算下，使用两种审计范式评估了五个前沿模型：基于VLA的探索后接基于VLM的异常识别，以及端到端VLM智能体（其中视觉推理直接指导动作选择）。在所评估的模型和两种范式中，成功率范围从6.6%到42.3%，远低于人类表现（83.4%）。通过世界审计任务，WorldAuditBench提供了一个测试平台，用于研究多模态智能体在交互式3D环境中如何耦合动作与视觉推理，同时突显了它们在探索过程中收集和解释证据能力的当前局限。
📄 English Abstract
As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomalies in these simulated environments, such as floating objects, traversable walls, or objects inconsistent with the surrounding scene. Multimodal AI systems, including vision-language models (VLMs) and vision-language-action models (VLAs), have shown potential for automating this task. However, 3D world auditing is complex, requiring the close coupling of two distinct capabilities: action, to navigate the 3D world and search for anomalies systematically and efficiently; and visual reasoning, to understand the environment and identify anomalies from multimodal observations. It remains largely unexplored whether multimodal agents can effectively couple these two capabilities, using visual reasoning to identify potential anomalies while taking actions to validate them. In this paper, we introduce WorldAuditBench, a benchmark for 3D world auditing comprising 213 anomaly tasks across 13 environments built with Unreal Engine 5 and Three.js, spanning five anomaly families. We evaluate five frontier models under a fixed exploration budget using two auditing paradigms: VLA-based exploration followed by VLM-based anomaly identification, and an end-to-end VLM agent in which visual reasoning directly guides action selection. Across the evaluated models and two paradigms, success rates range from 6.6% to 42.3%, substantially below human performance (83.4%). Through the task of world auditing, WorldAuditBench provides a testbed for studying how multimodal agents couple action and visual reasoning in interactive 3D environments, while highlighting current limitations in their ability to gather and interpret evidence during exploration.
8. 测试时 AI4AI 中面向智能体执行框架设计的元技能学习
- 英文标题：Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI
- arXiv：2609.38143
💡 一句话解读：这篇研究要解决的是：如何让一个AI（Builder）在模型参数不变的情况下，为另一个AI（Target）搭建更好的工作环境，从而提升其表现。方法是让Builder从Target的反馈中学习一套“元技能”——即何时该提供帮助、提供什么资源的原则，然后把这些原则用于新任务。实验表明，这种方法比没有技能或直接给技能效果更好，甚至同一模型兼任两角色也能自我提升。
智能体的表现既取决于推理能力，也取决于其行动的环境。我们研究测试时AI-for-AI，探讨在双方模型权重均固定的情况下，一个构建者（Builder）如何学会为目标模型（Target）构建更好的执行环境。为使构建者的经验可复用，我们引入元技能（Meta-Skill）：即规定何时需要支持以及提供何种资源的原则。构建者从目标模型在开发集上的执行反馈中学习这些原则，然后使用冻结的技能库为未见任务构建执行框架。在Harness-Bench和NewtonBench上，全库元技能相比无技能构建将宏平均性能提升了8.95个百分点，相比直接将同一技能库交给目标模型提升了12.02个百分点。这些结果凸显了将经验转化为可执行支持的价值。同一模型同时担任两个角色时也能获得提升，这进一步表明了一条通过学会构建更好环境来实现系统级自我改进的路径。
📄 English Abstract
Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a Builder can learn to construct better execution environments for a Target while both models' weights remain fixed. To make the Builder's experience reusable, we introduce Meta-Skill: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to system level self-improvement through learning to build better environments.
9. EVOKE：激发智能体中的世界知识以实现可迁移决策
- 英文标题：EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Making
- arXiv：2609.38334
💡 一句话解读：大语言模型做多步决策时，换到没见过的环境就表现很差。已有方法靠额外训练让模型预测未来，但代价高且误差会累积。这篇研究提出 EVOKE：不改变环境状态，而是让同一个状态面对多个不同目标，迫使模型改变动作偏好，从而把预训练时已记住的世界知识“逼”出来用于决策，提升了任务表现和迁移能力。
大语言模型（LLM）正越来越多地被用作多步决策的智能体，但在未见环境中的迁移能力很差。世界模型方法通过训练智能体预测未来观测来解决这一问题，代价是额外的训练成本，并且当预测被用于规划时误差会不断累积。然而，对于在数字环境中运行的 LLM 智能体而言，大量此类世界知识在预训练阶段就已被内化，这使问题从“获取知识”转变为“激发知识”。我们认为，典型的后训练几乎没有为这种激发提供压力，因为在每个访问到的状态下仅以单一目标进行监督，会无意中驱使策略依赖表层的上下文习惯。我们提出 EVOKE，一种通过固定状态下的目标多样性来施加这种压力的后训练方法。受理论启发——一个能在多样目标上都胜任的智能体，必然编码了一个可从其动作偏好中恢复的世界模型——EVOKE 保持环境状态和交互历史不变，并在不同目标下对同一组候选动作进行排序，迫使动作偏好发生变化，从而使依赖上下文习惯或单一目标相关性的策略无法正确排序。这隐式地激发了策略预训练的世界知识来指导决策。我们在三种骨干模型、多种任务上评估 EVOKE，证明了任务性能、未见环境泛化和数据效率方面的提升。我们进一步进行了受控分析，以更好地理解这些增益的来源。这些发现为通过直接决策监督来激发内化世界知识以实现可迁移动作提供了新视角。
📄 English Abstract
Large language models (LLMs) are increasingly deployed as agents for multi-step decision-making, yet transfer poorly to unseen environments. World-model methods address this by training agents to predict future observations, at the cost of additional training and errors that compound when predictions are used for planning. However, for LLM agents operating in digital environments, much of this world knowledge is already internalized during pretraining, which shifts the problem from acquiring it to eliciting it. We argue that typical post-training provides little pressure for such elicitation, since supervision under a single goal at each visited state inadvertently drives policies to rely on superficial contextual habits. We introduce EVOKE, a post-training method that supplies this pressure through goal diversity at fixed states. Motivated by theory showing that an agent competent across diverse goals must encode a world model recoverable from its action preferences, EVOKE holds the environment state and interaction history fixed and ranks the same candidate actions under alternative goals, forcing action preferences to change, so that a policy relying on contextual habits or single-goal correlations cannot order them correctly. This implicitly elicits the policy's pretrained world knowledge to inform decisions. We evaluate EVOKE across diverse tasks in three backbones, demonstrating improved task performance, unseen environment generalization, and data efficiency. We further conduct controlled analyses to better understand what drives these gains. These findings offer a new perspective on eliciting internalized world knowledge for transferable action through direct decision supervision.
10. RSIGame：具备递归自我改进能力的自主智能体游戏开发
- 英文标题：RSIGame: Autonomous Agentic Game Development with Recursive Self-improvement
- arXiv：2609.39045
💡 一句话解读：让 AI 自动做游戏时，反复修改容易只针对少数测试例子，做出漏洞多、玩法残缺的脆弱游戏。RSIGame 用「局部探索—诊断—改进」加「全局质量监控」的双层循环，边测边改并保留最佳版本，还把成功经验训练进模型。在 140 个任务上，它稳定提升游戏质量，并让 Qwen 模型以少 11 倍的生成量超过 GPT-5.5 的一次性成绩。
近期大语言模型的进展使自动游戏生成日益可行，但要在可玩版本之上可靠地提升生成游戏的质量仍具挑战。朴素的迭代式改进容易过拟合于一小组测试用例，产生漏洞未解、行为缺失、对更广泛玩家交互泛化性差的脆弱游戏。我们提出 RSIGame，一个具备递归自我改进能力的自主智能体游戏开发框架。RSIGame 将开发组织为互补的局部循环与全局循环：局部“探索—诊断—改进”循环广泛探索可执行游戏，诊断并排序所发现的问题，并进行有证据支撑的修订，其中不断演化的检查清单持续积累新的测试与改进指引；全局循环则跟踪整体质量、保留最佳检查点，并检测长时程开发中的饱和或退化。除测试时改进外，RSIGame 还通过训练将成功的开发经验内化到生成器中。在 140 个 GameCraft-Bench 任务、两个游戏引擎和五种生成器上，RSIGame 在相同开发预算下持续提升游戏质量。值得注意的是，经验内化使 Qwen3.8-27B 在 Godot 上达到 61.38、在 Phaser 上达到 58.53，超过 GPT-5.5 的一次性成绩，同时将 Qwen 的生成 token 数减少 11 倍。
📄 English Abstract
Recent advances in large language models have made automatic game generation increasingly feasible, yet reliably improving generated games beyond a playable version remains challenging. Naive iterative refinement can easily overfit a small set of test cases, producing fragile games with unresolved bugs, missing behaviors, and poor generalization to broader player interactions. We introduce RSIGame, an autonomous agentic game development framework with recursive self-improvement. RSIGame organizes development into complementary local and global loops. Concretely, a local explore-diagnose-improve loop broadly explores the executable game, diagnoses and prioritizes discovered issues, and performs evidence-grounded revision, where an evolving checklist continually accumulates new testing and improvement guidance. A global loop tracks overall quality, preserves the best checkpoint, and detects saturation or regression over long-horizon development. Beyond test-time improvement, RSIGame further internalizes successful development experience into the generator through training. Across 140 GameCraft-Bench tasks, two game engines, and five generators, RSIGame consistently improves game quality under matched development budgets. Notably, experience internalization enables Qwen3.8-27B to reach 61.38 on Godot and 58.53 on Phaser, exceeding GPT-5.5 one-shot scores while reducing Qwen's generation tokens by 11 times.
11. 更多选择，更少决策：类JEV直接决策模型中的序数尺度偏差
- 英文标题：More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models
- arXiv：2609.38827
💡 一句话解读：直接决策模型常被用来给文本打标签或评分，但用户给的评分等级（比如1到5分）模型往往用不全，总爱挤在中间几档。本文系统分析了JEV等模型，发现这种“等级压缩”现象普遍存在，且可通过后训练修正，并非模型天生缺陷。
直接决策模型（direct-decision models）能将文本转化为低延迟的结构化标签和分数，因此在分类和自动评估中颇具吸引力。然而，可靠性不仅要求准确率，还要求模型忠实使用用户提供的序数决策尺度（ordinal decision scale）。我们分析了JEV~1.13和三个开源KEV模型。研究从ANLI开始，JEV将38.8%的预测和51.3%的错误归为Neutral，尽管其准确率达74.95%、金标签近乎平衡且候选位置均衡。在36个序数数据集上，最终决策仅使用了67–76%的有效金支持（effective gold support），而在四个名义任务上为87–102%。随机化候选顺序会削弱但无法消除这种压缩。在固定项目和源分数、平衡金支持与位置的情况下，我们将尺度从K=2细化到14；所有模型的利用率均下降，在K=14时降至26–75%，尽管大多数模型的候选概率分布仍然较宽。针对性的BA-LoRA后训练在两种KEV规模下，将八个监督尺度上的金相对利用率从约47%提升至86%，表明这种压缩是可学习且可修改的，而非不可变的架构限制。我们将此称为序数尺度利用偏差（ordinal scale-utilization bias）：一种决策阶段的候选空间压缩，独立于准确率、金不平衡、固定位置和候选数量。代码和数据见https://github.com/Glax147/jev_ordinal_scale_bia。
📄 English Abstract
Direct-decision models turn text into low-latency structured labels and scores, making them attractive for classification and automatic evaluation. Yet reliability requires more than accuracy: a model must also use the ordinal decision scale supplied by the user faithfully. We analyze JEV~1.13 and three open KEV models. Our investigation begins with ANLI, where JEV assigns 38.8% of all predictions and 51.3% of errors to Neutral despite 74.95% accuracy, nearly balanced gold labels, and balanced candidate positions. Across 36 ordinal datasets, final decisions use only 67–76% of the effective gold support, versus 87–102% on four nominal tasks. Randomizing candidate order weakens but does not remove this compression. Holding items and source scores fixed while balancing gold support and positions, we refine scales from $K=2$ to $14$; utilization falls for every model and reaches 26–75% at $K=14$, although candidate probabilities remain broad for most models. Targeted BA-LoRA post-training raises gold-relative utilization from roughly 47% to 86% on eight supervised scales at both KEV sizes, showing that the compression is learned and modifiable rather than an immutable architectural limit. We call this ordinal scale-utilization bias: decision-stage candidate-space compression distinct from accuracy, gold imbalance, fixed position, and candidate count alone. The code and data are available at https://github.com/Glax147/jev_ordinal_scale_bia
12. Imagine3D-LLM：教会多模态大语言模型在回答前想象3D场景
- 英文标题：Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
- arXiv：2609.38177
💡 一句话解读：多视角图像的三维推理对多模态大模型仍很困难。本文提出Imagine3D-LLM，让模型在回答前先根据多视角图像“脑补”出一个粗略的三维场景（用3D高斯表示），再基于这个内部想象来作答，从而提升空间推理能力。
从多视角图像进行三维世界推理，对多模态大语言模型（MLLMs）而言仍是一项根本性挑战。尽管现代MLLMs能有效处理单图像输入，却难以跨视角整合证据以形成连贯的三维理解。已有工作试图通过增强细粒度像素级跨视角对应，或融合三维几何基础模型的特征来弥补这一差距，但与人类推理相比仍存在显著鸿沟。本文重新审视人类空间推理：人类并非依赖细粒度几何线索，而是大致识别跨视角的共同物体、推断视角间的相对几何关系，并组装出场景的粗略三维布局。受此启发，我们提出Imagine3D-LLM，一种学习组装类似紧凑三维场景表示并以此表示作为回答条件的大语言模型。具体而言，我们在图像标记后附加一小组可学习的摘要标记，将其解码为紧凑的三维高斯泼溅（3D Gaussian Splatting）表示，并以光度重建损失进行监督，同时与标准的下一词预测目标联合训练。值得注意的是，尽管只有摘要标记接受直接重建监督，该目标仍在LLM底层图像特征中诱导出更强的跨帧对应，表明学习重建会将三维感知信号传播至整个模型。因此，Imagine3D-LLM在多个空间推理和三维理解基准上持续优于先前方法，表明想象场景可能比被告知像素级几何更有效。
📄 English Abstract
Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.
13. 智能体错误数据集：面向失败分析与错误感知后训练，规模化构建5万条错误—诊断配对
- 英文标题：Agent Error Dataset: Scaling 50,000 Error–Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training
- arXiv：2609.40111
💡 一句话解读：这篇论文要解决的是：AI智能体（如网页操作助手）失败后，我们只看到最终失败结果，却不知道具体哪一步决策错了、该怎么改。作者构建了一个包含5万多条“错误-诊断”配对的数据集，并设计了一套五阶段流程，自动收集失败、生成诊断和修正建议，再用记录的证据验证。实验表明，用这些修正建议能大幅提高验证通过率，还能提升模型与教师标签的一致性。
一次不成功的LLM智能体（LLM agent）运行所包含的信息远不止其最终奖励：还包括智能体可获得的观察、它选择的动作以及环境的响应。要复用这些经验进行学习，需要识别出一个待修正的决策并测试一个具体的替代方案。我们引入了智能体错误数据集（Agent Error Dataset, AED），包含来自33个环境、19个执行框架家族和23个策略模型的9,961个源任务的50,228个错误-诊断配对，这些均基于文本智能体系统。我们保留了源轨迹和执行元数据，以支持跨设置的失败分析和重新诊断，而无需重复原始运行。我们的五阶段智能体错误到训练（Agentic Error-to-Training, AET）流程收集自然失败、生成诊断和提出的修正，并对照记录的证据进行检查。在支持重放的情况下，我们在匹配的执行设置下，将修正与来自同一检查点的原始动作重试进行比较。然后，我们为诊断和行动者恢复构建了单独的训练视图。在3,062个匹配的重放对中，首次提议的修正将验证器通过率从18.4%提高到51.1%，提升了32.7个百分点。使用单独冻结的诊断版本，在1,656个源任务上进行全诊断微调，将Qwen3-8B与内部教师标签的精确步骤一致性从47.2%提高到63.6%，这是在943个案例的保留集上三个种子的平均值。该比较中最强的提示参考得分为54.7%，并且平均一致性在四个递增的训练集大小上均有所提高。在行动者训练配方的单种子比较中，仅动作修复训练在WebShop-lite上比仅成功训练高出6.67个百分点。
📄 English Abstract
An unsuccessful LLM agent rollout contains more information than its final reward: the observations available to the agent, the actions it chose, and the environment's responses. Reusing this experience for learning requires identifying a decision to revise and testing a concrete alternative. We introduce the Agent Error Dataset (AED), comprising 50,228 error-diagnosis pairs from 9,961 source tasks across 33 environments, 19 harness families, and 23 policy models in text-based agent systems. We retain source traces and execution metadata to support cross-setting failure analysis and re-diagnosis without repeating the original rollout. Our five-stage Agentic Error-to-Training (AET) pipeline collects natural failures, generates diagnoses and proposed corrections, and checks them against recorded evidence. Where replay is supported, we compare corrections with original-action retries from the same checkpoint under matched execution settings. We then construct separate training views for diagnosis and actor recovery. Across 3,062 matched replay pairs, first-proposal corrections raise verifier pass rates from 18.4% to 51.1%, a gain of 32.7 percentage points. Using a separately frozen diagnosis release, full-diagnosis fine-tuning on 1,656 source tasks raises Qwen3-8B's exact-step agreement with internal teacher labels from 47.2% to 63.6%, averaged over three seeds on a 943-case holdout. The strongest prompted reference in this comparison scores 54.7%, and mean agreement improves at each of four increasing training-set sizes. In a single-seed comparison of actor-training recipes, action-only repair training scores 6.67 percentage points higher on WebShop-lite than success-only training.
14. LANTERN：照亮语言模型中的隐藏数学知识
- 英文标题：LANTERN: Illuminating Hidden Mathematical Knowledge in Language Models
- arXiv：2609.32264
💡 一句话解读：语言模型能证明定理，但找研究方向还得靠人。本研究用模型内部表示来发现数学中有价值的联系。方法叫LANTERN，用分类器对预训练模型激活排序候选关系，再经筛选、假设生成、可执行验证和分析检查。在整数序列百科上，从5000万对中找出62个已验证关系，其中4个全新。
语言模型如今能够证明定理，但决定研究哪些问题仍然由人来做。我们探究模型的内部表示是否有助于识别有前景的数学联系。我们开发了LANTERN，一个快速、成本高效的流程，它使用基于预训练模型激活的分类器对候选关系进行排序，随后进行分阶段过滤、假设生成、可执行验证和分析检查。将其应用于整数序列在线百科全书（OEIS），LANTERN在10,000个高频引用序列中对5000万对序列进行了排序，并在没有现有OEIS交叉引用的序列对之间产生了62个已验证关系。内容筛选保留了13个值得展示的关系；其中9个具有信息量或洞察力，包括4个据我们所知完全新颖的关系：这些关系既未出现在OEIS中，也未出现在我们的针对性文献检索中。整个端到端过程（包括分类器训练、候选排序、过滤和验证）耗时不到8小时。
📄 English Abstract
Language models can now prove theorems, but people still decide which problems to pursue. We ask whether a model's internal representations can help identify promising mathematical connections. We develop LANTERN, a fast, cost-efficient pipeline that uses a classifier over pretrained-model activations to rank candidate relations, followed by staged filtering, hypothesis generation, executable verification, and analytical checking. Applied to the On-Line Encyclopedia of Integer Sequences (OEIS), LANTERN ranked 50 million pairs among 10,000 frequently referenced sequences and produced 62 verified relations between pairs without an existing OEIS cross-reference. A content screen retained 13 relations worth presenting; nine of these are informative or insightful, including four which are entirely novel to the best of our knowledge: none appears in the OEIS or in our targeted literature search. The entire end-to-end process including classifier training, candidate ranking, filtering and verification took under 8 hours.
15. 并非图像所示：无关上下文会扰乱 VLM 评判模型却不为其提供信息
- 英文标题：It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
- arXiv：2609.37863
💡 一句话解读：这篇研究要解决的是：用视觉语言模型（VLM）替代人工标注时，测试结果是否可靠。他们设计了MIST压力测试，用200个可作比喻或字面理解的句子，分别配上一致、误导或无图片，要求仅根据句子判断。结果发现，图片的存在本身就会改变模型的判断，而非图片内容，说明替代性结论更多反映的是测试配置而非模型能力。
视觉语言模型（VLM）正越来越多地替代人类标注者，因此替代性测试必须反映模型本身，而非偶然的评估条件。我们提出了MIST（误导性图像压力测试）：包含200个英文句子，每个句子围绕一个可作比喻或字面理解的短语，并配以一致图像（描绘其字面义）、误导图像（描绘相反义）或无图像。指南要求仅根据句子决定标签，因此任何图像都不应改变答案。我们预期每种图像都会将评判者的标签拉向其描绘的含义，但两种图像均未产生此效果。在13个VLM评判者中，一致图像改变了20.5%的标签，误导图像改变了19.4%，每个评判者的这两个比例都接近，且均高于在保留图像但删除“忽略图像”指令时产生的11.6%。然而，两种图像间有差异的标签中，只有37%转向了所展示的含义，且无论图像缺失、一致还是误导，与人类标注者的一致性均无变化。在通过替代测试的7个评判者中，该效应小于从未通过的6个评判者，但在所有评判者中均存在：驱动评判者的是图像的存在，而非两种图像中的哪一种，因此替代性结论既描述模型，也描述配置。
📄 English Abstract
Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions. We introduce MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all. The guidelines require the label to be decided from the sentence alone, so no image should change any answer. We expected each image to pull a judge's labels toward the sense it depicts, and neither kind did. Across thirteen VLM judges, an aligned image changed 20.5% of labels and a misleading one 19.4%, close for every judge and both above the 11.6% produced by deleting the ignore-the-image instruction with the image left in place. Yet only 37% of the labels that differ between the two images moved toward the sense shown, and agreement with our human annotators is unchanged whether the image is absent, aligned or misleading. The effect is smaller in the seven judges that pass the alt-test than in the six that never do, but present in all of them: what moves a judge is that an image is there, not which of the two it is, so a substitutability verdict describes a configuration as much as a model.