Daily Papers · 2026-09-23
📄 论文日报 | 来源：Hugging Face Daily Papers
🗓 Episode：2026.09.22 | VLM智能迁移至机器人控制；隐式3D记忆构建视频世界模型
1. 将VLM的智能迁移至机器人控制
- 英文标题：Transferring the Intelligence of VLMs to Robotic Control
- arXiv：2609.22966
💡 一句话解读：这篇研究想解决的是：能否让原本只在数字世界工作的视觉语言模型，直接用来控制真实机器人。作者设计了一个简单接口，把机器人动作变成平移、旋转和夹爪开合等离散指令，让模型边看边做、闭环调整，并用少量示范教它上手。实验表明，无需专门训练就能超过不少专用方法，加一个示范后效果更好，还能迁移到真实机器人。
人类能够无缝适应物理世界和数字世界，这表明尽管在具身、环境和任务上存在从数字到现实的差距，人类智能本身或许可以跨越这一差距迁移。这自然引出一个根本问题：视觉语言模型（VLM）的智能能否同样从数字世界泛化到物理世界，用于机器人控制？我们通过 RoboDawn 研究这一问题。RoboDawn 是一种符合人类直觉的接口，它通过一组紧凑的离散平移、旋转和夹爪命令，将机器人控制暴露给一个具备智能体能力的 VLM。借助该接口，VLM 以闭环方式控制机器人：它观察当前视觉状态，推理下一步动作，执行该动作，并根据由此产生的状态调整后续决策。此外，我们引入了一种上下文学习（ICL）方案，利用少量示范让 VLM 同时掌握接口使用和任务求解策略。在 RoboTwin 2.0 C2R 和 RoboDojo 上的实验表明，RoboDawn 无需针对特定任务进行机器人训练即可取得强劲性能。在零样本设置下，RoboDawn 优于多个在基准专用机器人数据上训练的强大策略；而单个上下文示范进一步带来显著性能提升，并确立了当前最优（SOTA）结果。在 RoboTwin 2.0 C2R 上，成功率从零样本的 53.2% 提升到单样本的 73.6%，超过了稳健基线 π0.5（46.0%）。在 RoboDojo 上也观察到类似增益，成功率从零样本的 35.67% 提升到单样本的 47.17%。同一框架还可迁移到真实机器人，在 Franka 上执行把方块放入篮子和堆叠方块任务。
📄 English Abstract
Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results. On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline {\pi}0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.
2. WorldCrafter：具有隐式3D感知记忆的一致视频世界模型
- 英文标题：WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
- arXiv：2609.24984v1
💡 一句话解读：视频世界模型在长时间探索时容易忘记之前看到的场景，换个视角就对不上。WorldCrafter 让模型学会一种“隐式3D记忆”：根据当前要看的视角，把历史画面压缩成少量关键信息，再生成新画面，从而在长达一分钟的探索中保持场景一致和镜头控制准确。
视频世界模型能够对动态环境进行交互式探索，但难以在长时程和跨视角下保持与先前观测的一致性。我们提出 WorldCrafter，一种视频世界模型，它为此学习了一种可查询相机的隐式3D感知记忆。其核心思想是让请求的视角决定多视角证据如何被压缩到视频生成器有限的 token 预算中。记忆编码器和姿态条件读出模块与视频生成器联合训练，在去噪前将历史观测整合为一组固定的目标视角特定 token，而无需显式的基于深度的对应关系。通过将该记忆与近期时间上下文和少步蒸馏相结合，WorldCrafter 能够从单张输入图像或文本提示进行流式场景探索。在静态和动态场景上的实验表明，该方法在长时程一致性和相机控制精度上取得显著提升，同时在分钟级探索中保持视觉质量。
📄 English Abstract
Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator’s limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.
3. GameHorizon 套件：游戏玩法中的多时间跨度数据与评估
- 英文标题：GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
- arXiv：2609.25001v1
💡 一句话解读：现有游戏AI测试要么游戏种类少、缺语言指令，要么依赖高波动的在线试玩，难以公平比较。作者推出GameHorizon套件：自动标注多时间跨度指令的流水线、5000小时21款3A游戏的人类专家数据，以及可复现的离线与在线测试基准，并用它评测了47个模型。
现代电子游戏为AI模型提供了一个可量化的测试平台，它综合考察视觉理解、指令分解、目标规划以及在多个时间跨度上的精确动作控制能力。然而，现有数据集和基准要么覆盖的游戏范围狭窄，要么缺少语言指令，要么依赖高方差的在线试玩。为应对这些挑战，我们提出了GameHorizon——一个统一的数据与评估套件，用于衡量不同模型家族在不同时间跨度上的游戏能力。GameHorizon套件包含三个部分：第一，GameHorizon-Annotator，一个可扩展、自动化的多时间跨度指令标注流水线；第二，利用该流水线，我们构建了GameHorizon-Data，这是首个大规模3A游戏数据集，包含时间对齐的视频、玩家动作和多时间跨度指令，由100名人类专家玩家采集了21款游戏共5000小时的录像；第三，我们构建了GameHorizon-Bench，提供可复现的离线测试和逐步在线测试。离线赛道使用数千个标准化问题（组织为三个主要任务及一系列诊断变体）进行可复现评估，在线赛道则检验离线分数是否反映实际游戏能力，并将失败定位到长时程游戏中的具体步骤。基于GameHorizon套件，我们通过超过一百万次模型调用评估了47个模型，揭示了任务难度的有意义的层级以及模型能力之间的显著差异。我们的工作可为跨时间跨度和模型家族的游戏能力评估提供标准化标尺。我们将发布数据集、标注器和基准，以促进未来研究。
📄 English Abstract
Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, and precise action control over multiple temporal horizons. Existing datasets and benchmarks, however, either cover a narrow range of games, lack language instructions, or rely on high-variance online rollouts. To address these challenges, we introduce GameHorizon, a unified data and evaluation suite that measures gameplay capabilities at different horizons for diverse model families. GameHorizon Suite consists of three components. First, GameHorizon-Annotator is a scalable and automated annotation pipeline for multi-horizon instructions. Second, utilizing the pipeline, we construct GameHorizon-Data, the first large-scale AAA gameplay dataset with temporally aligned videos, player actions, and multi-horizon instructions. It comprises 5,000 hours of recordings from 21 games, collected by 100 human expert players. Third, we build GameHorizon-Bench with reproducible offline and stepwise online testing. The offline track enables reproducible evaluation using thousands of standardized questions organized into three primary tasks and a series of diagnostic variants, while the online track tests whether offline scores reflect actual gameplay capabilities and localizes failures to specific steps within long-horizon gameplay. Based on our GameHorizon Suite, we evaluate 47 models through more than one million model invocations, revealing a meaningful hierarchy of task difficulty and pronounced differences in model capabilities. Our work can provide a standardized yardstick for evaluating gameplay capabilities across horizons and model families. We will release our dataset, annotator, and benchmark to facilitate future research.
4. RRSI：智能体运行框架的正则化递归自我改进
- 英文标题：RRSI: Regularized Recursive Self-Improvement of Agent Harnesses
- arXiv：2609.24972v1
💡 一句话解读：大模型智能体的能力很大程度取决于它外部的“装备”（提示词、流程、工具、记忆等）。已有方法让这套装备自动迭代升级，但容易死记训练任务，换个场景就不灵。本文给升级过程加上“约束”：限制每次改动幅度、鼓励探索新方向、筛掉只对特定测试有效的改动。结果在多个任务上更稳、更省。
大语言模型智能体的能力在很大程度上被其“运行框架”（harness）所放大，即围绕冻结的主干模型的提示词、控制流、工具、记忆与上下文管理。近期方法通过迭代地提出并选择对智能体运行框架的组件级修改，日益自动化这一过程，实际上在智能体系统层面建立了一种递归自我改进（RSI）。然而，这种递归演化可能因记忆训练任务而过拟合，在分布内表现出大幅提升，但在分布外基准上收益缩小甚至消失。我们提出“智能体运行框架的正则化递归自我改进”（RRSI），通过约束演化候选的提出与选择，将正则化原则引入运行框架的自我改进。提出器以随时间退火的预算运行，限制单个候选可捆绑的修改数量，并基于演化历史鼓励探索未走过的轨迹。选择器配备一个评论器（critic）和一个剪枝器（pruner）：评论器筛除针对特定基准的提案，剪枝器则移除过小、代价过高或不再有用的改动。这些约束共同倾向于可复用的智能体机制，而非特定于基准的机制甚至噪声。在涵盖编程、智能体工作空间与工程设计任务的八个基准上，RRSI 在其演化所针对的数据划分上最高提升 14.1 分，在五个分布外基准上最高提升 4.7 分，同时生成的运行框架相比未正则化的演化少用 30% 的策略 token。代码见 https://github.com/google-research/rrsi，项目主页为 https://regularized-rsi.com/。
📄 English Abstract
An LLM agent’s capability is largely magnified by its harness, namely the prompts, control flow, tooling, memory, and context management surrounding the frozen backbone model. Recent methods increasingly automate this process by iteratively proposing and selecting component-wise edits of an agent harness, practically establishing a form of recursive self-improvement (RSI) at the agent-system level. However, such recursive evolution may overfit by memorizing the training tasks, showing large in-distribution gains that shrink or even vanish on out-of-distribution benchmarks. We introduce Regularized Recursive Self-Improvement of Agent Harnesses (RRSI), which incorporates the principles of regularizations into harness self-improvement by constraining the evolution candidate proposal and selection. The proposer operates with a temporally annealed budget, limiting how many edits a candidate can bundle, and it encourages unexplored trajectories based on evolution history. The selector is equipped with a critic and a pruner: the critic screens benchmark-specific proposals, while the pruner, removes changes that are too small, too expensive, or no longer useful. Together these constraints favor reusable agent mechanisms over benchmark-specific ones or even noises. Across eight benchmarks spanning coding, agentic workspace and engineering design tasks, RRSI gains up to 14.1 points on the split it evolves against and up to 4.7 points on the five out-of-distribution benchmarks, while producing a harness that runs on 30% fewer policy tokens than the unregularized evolution. Code is available at https://github.com/google-research/rrsi and project page is https://regularized-rsi.com/.
5. 文档检索感知分块（D-RAC）：通过 PDF 规范化与多模态 Markdown 转换实现企业文档的通用检索感知摄取
- 英文标题：Document Retrieval-Aware Chunking (D-RAC): Universal Retrieval-Aware Ingestion of Enterprise Documents via PDF Normalization and Multimodal Markdown Conversion
- arXiv：2609.24220v1
💡 一句话解读：企业知识库里的文档格式五花八门，内容常被复杂排版锁住，现有方法要么把表格和标题层级弄乱，要么用大模型切分太贵还容易瞎编。D-RAC 先把任何文档转成 PDF，再用一次多模态大模型把页面转成便于检索的 Markdown，最后按标识符确定性地切块，成本大降、速度更快。
面向企业知识库的检索增强生成（RAG）系统必须摄取异构文档格式——PDF、Word 文档、演示文稿和扫描件——其内容被锁在复杂的视觉布局、多栏页面和密集表格中。基于规则的抽取和 OCR 会破坏阅读顺序、压平表格并丢失标题层级，而对抽取文本进行完全智能体式（agentic）分块则带来高昂的 token 成本和幻觉风险。我们提出文档检索感知分块（D-RAC），将我们的 Web 检索感知分块（W-RAC）框架扩展到任意文档格式。D-RAC 首先将任意输入文档规范化为 PDF，利用几乎每种格式都有忠实、确定性 PDF 渲染这一事实。随后一次多模态 LLM 处理将渲染页面转换为面向检索优化的 Markdown——把表格改写为自包含的散文陈述并保留标题层级——之后分块完全按照 W-RAC 进行：确定性地解析为可被 ID 寻址的单元，再基于标识符而非文本进行轻量级 LLM 分块规划。分块过程中从不重新生成源文本，从而保留 W-RAC 的成本、确定性和可观测性优势，同时将每种可渲染格式解锁为一等输入。在涵盖五个企业领域的 RAG-Multi-Corpus 基准的 236 份文档、795 页 PDF 子集上，D-RAC 在 72 分钟内以零错误转换并分块整个语料库，产生 1,748 个可供检索的块。与使用前沿 LLM 的智能体式分块相比，D-RAC 将分块阶段输出 token 减少 95.7%，将分块成本降低 77.8%（GPT-4.1 定价）至 85.6%（Gemini 2.5 Pro 定价），并将分块时间缩短 75%。D-RAC 可线性扩展到 500 页以上的文档。
📄 English Abstract
Retrieval-Augmented Generation (RAG) systems over enterprise knowledge bases must ingest heterogeneous document formats – PDFs, Word documents, presentations, and scans – whose content is locked inside complex visual layouts, multi-column pages, and dense tables. Rule-based extraction and OCR destroy reading order, flatten tables, and lose heading hierarchy, while fully agentic chunking over extracted text incurs high token costs and hallucination risk. We present Document Retrieval-Aware Chunking (D-RAC), an extension of our Web Retrieval-Aware Chunking (W-RAC) framework to arbitrary document formats. D-RAC first normalizes any input document into PDF, exploiting the fact that virtually every format has a faithful, deterministic PDF rendering. A single multimodal LLM pass then converts rendered pages into retrieval-optimized Markdown – rewriting tables as self-contained prose statements and preserving heading hierarchy – after which chunking proceeds exactly as in W-RAC: deterministic parsing into ID-addressable units followed by lightweight LLM-based chunk planning over identifiers rather than text. Source text is never regenerated during chunking, preserving W-RAC’s cost, determinism, and observability benefits while unlocking every renderable format as a first-class input. On the 236-document, 795-page PDF subset of the RAG-Multi-Corpus benchmark spanning five enterprise domains, D-RAC converts and chunks the entire corpus in 72 minutes with zero errors, producing 1,748 retrieval-ready chunks. Compared to agentic chunking with frontier LLMs, D-RAC reduces chunking-stage output tokens by 95.7%, cutting chunking cost by 77.8% (GPT-4.1 pricing) to 85.6% (Gemini 2.5 Pro pricing) and chunking time by 75%. D-RAC scales linearly to documents of 500+ pages.
6. OmniEdu：面向学习与教学的开放基础模型
- 英文标题：OmniEdu: Open Foundation Models for Learning and Teaching
- arXiv：2609.23088v1
💡 一句话解读：现有教育AI要么只会解题，要么只会辅导，训练数据按来源堆砌而非按能力组织。OmniEdu把数据按学科能力、课程理解、诊断推理、教学支持四种能力重新整理，训练出4B到27B的模型，在解题、课程和辅导测试中全面领先。
教育基础模型必须能够解决问题、理解课程结构、诊断学习困难并提供适当的教学支持。现有的教育语言模型通常只侧重解题或辅导，其训练数据混合按来源或任务组织，而非按能力组织。我们提出了OmniEdu，一个面向K-12学习与教学的开放基础模型系列。其指令微调语料库整合了100多种教育资源与通用指令来源，围绕四种能力组织：学科能力、课程基础、诊断推理以及教学行动与支架。我们的流程集成了确定性清洗、语义审计与重写、任务特定质量评分、令牌预算多样性选择以及教学指令分配，最终生成69,999个示例和1596万个监督响应令牌，其中包含60,951个教育专用示例。我们微调了4B、9B和27B模型，并评估了课程基础、K-12问题解决和教学辅导能力，以及通用能力。面向教育的调优在所有模型规模上一致提升了三个教育基准组的表现。OmniEdu-27B在K12-Bench上达到63.12%的EM和76.69%的F1，在MathFish上达到85.89%，在EDUMATH上达到86.95%，在MathTutorBench的支架设置中达到78.74%。在评估模型中，它在LongTutor上取得了最高的教学平均分3.02。这些结果证明了经过精心策划、能力均衡的监督对于将通用语言模型适配到涵盖问题解决、课程理解和教学支持的教育任务中的价值。
📄 English Abstract
Educational foundation models must solve problems, understand curriculum structure, diagnose learner difficulties, and provide appropriate instructional support. Existing educational language models often focus on either problem solving or tutoring, with training mixtures organized by source or task rather than capability. We present OmniEdu, an open family of foundation models for K-12 learning and teaching. Its instruction-tuning corpus combines over 100 educational resources and general instruction sources, organized around four capabilities: subject competence, curriculum grounding, diagnostic reasoning, and pedagogical action and scaffolding. Our pipeline integrates deterministic cleaning, semantic auditing and rewriting, task-specific quality scoring, token-budgeted diversity selection, and pedagogical instruction assignment. It yields 69,999 examples and 15.96M supervised response tokens, including 60,951 education-specific examples. We fine-tune 4B, 9B, and 27B models and evaluate curriculum grounding, K-12 problem solving, and pedagogical tutoring, alongside general capability. Education-oriented tuning consistently improves all three educational benchmark groups across model scales. OmniEdu-27B achieves 63.12% EM and 76.69% F1 on K12-Bench, 85.89% on MathFish, 86.95% on EDUMATH, and 78.74% in MathTutorBench’s Scaffold setting. It also achieves the highest Teaching average on LongTutor among the evaluated models, at 3.02. These results demonstrate the value of curated, capability-balanced supervision for adapting general language models to educational tasks spanning problem solving, curriculum understanding, and instructional support.
7. VideoGen-Agent：强化视频生成智能体
- 英文标题：VideoGen-Agent: Reinforcing Video Generation Agents
- arXiv：2609.24997v1
💡 一句话解读：现有视频生成模型遇到需要专业知识、特定人物、物理规律或先后顺序的提示词时常常做不好。本文提出 VideoGen-Agent，一个会调用外部工具的多模态智能体，通过多轮交互协调增强、生成和验证工具，并用强化学习训练它学会用工具，从而显著提升生成质量。
近期视频生成模型的进展已能实现高保真、时间连贯的视频生成。然而，这些模型往往难以满足需要专业知识、特定身份、物理一致性或有序事件的提示词。本文提出 VideoGen-Agent，一个通过多任务智能体强化学习训练的多模态智能体，利用外部工具进行视频生成。该智能体通过多轮交互协调增强、生成和验证工具，并依据提示词与中间观察来指导决策。我们在涵盖六类任务的类别均衡数据集上训练一个共享策略：先在教师生成的轨迹上进行监督微调以建立工具使用行为，再通过强化学习进行优化；同时采用类别感知的混合奖励来评估工具调用的有效性、任务适配的工具使用以及生成视频的质量。我们还提出了 VABench，一个包含 600 条提示词的留出基准，覆盖程序性知识、单实体与多实体身份保持、物理一致性、场景构图以及多镜头时间结构。在 VABench 上，VideoGen-Agent 比其基础文本到视频生成器提升了 19.1 分，从 56.5 提升至 75.6；升级生成工具后，无需额外智能体训练即可将分数进一步提升至 86.1。在 84.3% 的对比中，人类评分者更偏好升级后的配置，而非最强的独立基线。这些结果支持跨视频生成任务学习工具使用，并表明训练后的智能体能够受益于生成工具的后续进步。
📄 English Abstract
Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.
8. onPanda：通过Token级纠正为LLM与智能体高效标注同策略对齐数据
- 英文标题：onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction
- arXiv：2609.24983v1
💡 一句话解读：这篇论文要解决的是：人工标注大模型对齐数据时，逐字修改模型回答既费时又容易让数据偏离模型原本的说话方式。为此它做了个叫onPanda的工具，标注者只需找到第一个不对的地方，改掉它，然后让模型接着往下写，如此反复，省时一半以上，还能顺便留下精细的对错样本。
我们提出了onPanda，一个用于高效标注大语言模型（LLM）对齐数据和智能体轨迹的交互式工具。onPanda以Token级纠正为核心交互方式：标注者在阅读模型回答时，定位第一个不合适的Token，然后从模型的候选Token中选择替代项，或通过自由编辑输入正确文本。系统随后截断该位置之后的所有内容，并从纠正后的前缀继续生成，重复这一“定位—纠正—继续”循环，直到获得满意的回答。这种机制让标注者能以低成本精确引导模型输出：一项小规模对照研究表明，onPanda将标注时间中位数比人工后期编辑减少了52%。由于最终回答中绝大多数Token由模型自身生成，所得数据在很大程度上保留了模型的采样分布，非常适合构建同策略（on-policy）的SFT和偏好数据。此外，标注过程中记录的Token级纠正提供了具有精确位置和自然配对的正负样本的细粒度监督。onPanda还可连接外部工具和运行环境，支持在真实环境中进行交互式轨迹标注。我们还发布了用onPanda标注的数据集Panda-CVL，以及一个Token级纠正的基准测试。
📄 English Abstract
We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model’s candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained. This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing. Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model’s sampling distribution and is well suited for constructing on-policy SFT and preference data. Furthermore, the token-level corrections recorded during annotation provide fine-grained supervision with precise positions and naturally paired positive–negative samples. onPanda also connects to external tools and harnesses, enabling interactive trajectory annotation in realistic environments. In addition, we release Panda-CVL, a dataset annotated with onPanda, together with a benchmark for token-level correction.
9. 接地动作模型：以3D接地作为机器人学基础
- 英文标题：Grounded Action Model: 3D Grounding as a Foundation for Robotics
- arXiv：2609.23863v1
💡 一句话解读：机器人要抓东西，得先知道目标是什么、在哪里，但现有模型只靠看视频或文字来猜，容易出错。这篇论文提出“接地动作模型”，用3D信息直接锁定物体，让机器人更准地完成操作。
操作策略必须知道哪些物体重要以及它们在哪里，然而当前机器人基础模型所依赖的预训练骨干网络——从视觉-语言-动作模型（VLAs）中的语言到世界-动作模型（WAMs）中的视频生成——并不直接要求这种度量接地，而是将其留给从机器人演示中隐式学习。我们提出接地动作模型（GAMs），这是一种以3D接地构建的机器人基础模型新范式。GAM可以接受语言、点或框提示作为条件，这些提示首先被转换为所选对象的共享对象中心表示。该表示捕获了目标聚焦的视觉特征和度量对象几何，并通过多流Transformer与机器人状态历史混合，以预测动作块。尽管GAM可以自主运行，它也可以作为低级控制器，由高级规划器使用其各种输入模态进行控制，从而实现长时程和依赖记忆的操作。在RoboTwin 2.0上，GAM在50个任务中平均成功率达到55.3%（对比Spatial Forcing的52.0%），其中在场景随机化下为47.6%（对比Abot-M0的30.4%），而其动作策略仅在干净场景演示上训练。在LIBERO-PRO上，它在16种扰动设置中实现了最先进的平均成功率61%（对比π_{0.5}的53%），在目标被重新定位或新指定时增益最大。在两台真实机器人上，GAM在双臂YAM上视觉偏移下保持17/20的成功率，而π_{0.5}为4/20；同时，其与Molmo2规划器在Franka上的组合在长时程和依赖记忆的任务上实现了64.7%的ID和49.8%的OOD步骤完成率。
📄 English Abstract
Manipulation policies must know which objects matter and where they are, yet the pretrained backbones that current robot foundation models build on, from language in vision-language-action models (VLAs) to video generation in world-action models (WAMs), do not directly require this metric grounding, leaving it to be learned implicitly from robot demonstrations. We propose Grounded Action Models (GAMs), a new paradigm of robot foundation models built with 3D grounding. GAM can be conditioned using language, points, or box prompts, which are first transformed into a shared object-centric representation of the selected objects. This representation captures target-focused visual features and metric object geometry, which is mixed with robot state history through a multi-stream transformer to predict action chunks. Although GAMs can be run autonomously, they can also serve as a low-level controller that a high-level planner controls using its various input modalities, allowing for long-horizon and memory-dependent manipulation. On RoboTwin 2.0, GAM achieves an average success rate of 55.3% across 50 tasks (vs. 52.0% for Spatial Forcing), including 47.6% under scene randomization (vs. 30.4% for Abot-M0), with its action policy trained only on clean-scene demonstrations. On LIBERO-PRO, it achieves a state-of-the-art average success rate of 61% (vs. 53% for $π_{0.5}$) across 16 perturbation settings, with the largest gains when targets are relocated or newly designated. On two real robots, GAM retains 17/20 successes under visual shift on a bimanual YAM versus 4/20 for $π_{0.5}$, while its composition with a Molmo2 planner on a Franka achieves 64.7% ID and 49.8% OOD step completion on long-horizon and memory-dependent tasks.
10. 从一到多，从多到一：面向软件工程智能体的类别感知迭代专家训练
- 英文标题：One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents
- arXiv：2609.23377v1
💡 一句话解读：软件工程任务种类繁多，用同一种方法训练AI智能体时，常出现“按下葫芦浮起瓢”——某些任务变好、另一些反而变差。本研究提出“分门别类训练专家、再合并成一个模型”的框架：先按任务类别分别训练专家，再通过标签路由的蒸馏技术把它们整合成一个可部署的模型，最终在多个基准上取得明显提升。
仓库级软件工程（SWE）包含多种异构任务类别，在混合式智能体强化学习下，各类别的进展可能不均衡：某些类别的提升伴随着其他类别的退步，而总体解决率会掩盖这些变化。受这种“类别跷跷板”现象启发，我们提出了一种类别感知的专家训练与策略整合框架。可执行任务构建与 SWE Labeler（一个基于证据的多轴标注系统）负责组织训练池。初始的类别专属强化学习提升了平均训练成功率，但实例级进展仍不均衡，这促使我们显式地巩固成功行为并进行策略自适应任务选择。同源类别专家交替进行长时程 Agentic-miniRL 与刷新-修复-扩展（RRE）：更新后的策略刷新实例掌握情况，复用自身已验证的成功轨迹进行修复式监督微调（Repair SFT），并重新选择任务以进行进一步强化学习。标签路由的多教师同策略蒸馏（MOPD）将各专家整合为一个可部署的学生模型，并通过 ReLU 门控的奖励外推，仅保留每个教师相对于参考模型的改进方向。专家训练与策略整合均无需外部模型提供解题轨迹或动作目标。我们通过总体与分类别解决率、相对各联合强化学习基线的最小类别提升以及专家增益恢复率，评估了混合式强化学习与平衡式强化学习、专家开发以及单模型整合。最终 MOPD 策略在 Pro-618 上达到 58.04% 的平均解决率，在 SWE-bench Multilingual 上达到 59.00%，分别比基础模型提升 5.39 和 2.78 个百分点。
📄 English Abstract
Repository-level software engineering (SWE) comprises heterogeneous task categories, whose progress under pooled agentic reinforcement learning can be uneven: gains in some categories coincide with regressions in others, while aggregate resolution obscures these changes. Motivated by this category see-saw, we develop a category-aware expert-training and policy-integration framework. Executable task construction and SWE Labeler, an evidence-grounded multi-axis labeling system, organize the training pools. Initial category-specific RL improves average training success while leaving uneven instance-level progress, motivating explicit consolidation of successful behavior and policy-adaptive task selection. Same-origin category experts alternate long-horizon Agentic-miniRL with Refresh-Repair-Expand (RRE): the updated policy refreshes instance mastery, reuses its own verified successful trajectories for Repair SFT, and reselects tasks for further RL. Label-routed multi-teacher on-policy distillation (MOPD) consolidates the experts into one deployable student, with ReLU-gated reward extrapolation keeping only each teacher’s improving direction over the reference. Expert training and policy integration require no external model to provide solution trajectories or action targets. We evaluate Pooled RL and Balanced RL, expert development, and single-model integration through aggregate and per-category resolution, the minimum category lift over each joint-RL baseline, and expert-gain recovery. The final MOPD policy achieves mean resolution of 58.04% on Pro-618 and 59.00% on SWE-bench Multilingual, improving over the base model by 5.39 and 2.78 percentage points, respectively.
11. Deep Persona：面向角色扮演智能体与模拟的心理学基础架构与评估框架
- 英文标题：Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
- arXiv：2609.22255v1
💡 一句话解读：现有大语言模型角色扮演常因角色设定太浅，聊久了就人设崩塌。本文提出“深度角色”三层架构，把角色分为外在表现、潜在信念和核心动机，并限制模型按内部剧本反应，同时设计无参考评估框架，用心理量表和压力测试检验对话自然度。
现有基于大语言模型（LLM）的角色模拟方法大多依赖浅层的角色描述，无法在长时间交互中维持连贯的角色行为。我们提出 Deep Persona，一种以心理学为基础的三层架构，将角色组织为可观察表达、潜在信念和核心动机驱力三个层级，用于构建高度可信的角色扮演智能体。该架构遵循脚本化决定论和有限能动性原则，将模型限制为由结构化内部脚本引导的反应式引擎。我们进一步提出一种无参考评估框架，利用成熟的心理临床工具和对抗性压力测试，将对话自然度与人类经验分布进行基准比较。实证评估表明，尽管 LLM 实现了较高的语用流畅度，但在情感表达和共同注意方面表现出系统性局限。此外，我们展示了两个 Deep Persona 的案例研究，并使用所提出的框架进行评估，证明结构化角色能够产生更贴近人类对话行为的交互。
📄 English Abstract
Existing approaches to persona simulation with Large Language Models (LLMs) mostly rely on shallow character descriptions that fail to sustain coherent character behavior across extended interactions. We introduce Deep Persona, a psychologically grounded, three-layered architecture that organizes personas into hierarchical levels of observable expression, latent beliefs, and core motivational drives, for constructing highly convincing role-playing agents. Governed by the principles of scripted determinism and bounded agency, the architecture restricts the model to a reactive engine guided by a structured internal script. We further propose a reference-free evaluation framework that benchmarks dialogue naturalness against empirical human distributions using established psychological clinical instruments and adversarial stress-tests. Empirical evaluation reveals that while LLMs achieve high pragmatic fluency, they exhibit systematic limitations in emotional expression and joint attention. In addition, we present a case study of two Deep Personas and evaluate them using the proposed framework, demonstrating that structured personas can produce interactions that more closely align with human conversational behavior.
12. Harness-Zero：通过智能体作为外壳进行外壳蒸馏
- 英文标题：Harness-Zero: Harness Distillation via Agent-as-Harness
- arXiv：2609.24974v1
💡 一句话解读：不同任务、不同模型往往需要不同的“外部辅助系统”（外壳）来提升表现，但部署时只能固定用一个，难以兼顾。作者提出 Harness-Zero：让一个智能体充当“外壳”，在训练时把优化外壳的指导转化为修正后的示范轨迹，微调进模型权重，使这些好处在去掉外壳后依然保留。
智能体外壳（agent harness）——即协调模型与环境交互的外部系统——能显著提升智能体表现，但其收益在部署时仍受限于该外壳。由于最优外壳因领域、实例和模型而异，通用智能体要么只能接受次优的共享外壳，要么在日益增多的专用外壳之间进行路由。因此，我们研究智能体外壳蒸馏（agent harness distillation）：将针对领域或实例优化的外壳作为训练时的指导，把它所诱导的行为迁移到模型权重中，从而使其收益在单一固定目标外壳下依然保留。难点在于两种外壳在动作空间和可用信息上不同，因此优化外壳的指导无法直接作为目标外壳的监督信号。我们提出 Harness-Zero，通过“智能体作为外壳”（agent-as-harness）实现外壳蒸馏。在优化外壳的引导下，一个充当外壳的智能体会在执行前于目标外壳的动作空间中修正学生模型的回答，从而将外壳指导转化为训练示范。对由此产生的轨迹进行微调，可将外壳诱导的行为内化到模型中，因此部署时可以移除专用外壳。我们在知识工作、工具使用和科学领域开展的实验表明：(1) 对于使用同一演化外壳的前沿大语言模型，智能体作为外壳优于代码作为外壳（code-as-harness）。(2) 在部署时移除专用外壳的情况下，Harness-Zero 将基础模型的宏平均任务成功率从 23.3% 提升至 44.3%，甚至超过仍挂载该外壳时的 41.7%。(3) Harness-Zero 恢复了基础模型中缺失的外壳诱导行为，在三个领域的 28 种模式上平均恢复率达 82.3%。
📄 English Abstract
Agent harnesses, the external systems that mediate model-environment interaction, can substantially improve agent performance, but their gains remain tied to the harness at deployment. Because the best harness varies across domains, instances, and models, a general-purpose agent must either settle for a suboptimal shared harness or route among an ever-growing set of specialized ones. We therefore study agent harness distillation: using a domain- or instance-optimized harness as training-time guidance and transferring the behaviors it induces into model weights, so that its gains survive under a single fixed target harness. The challenge is that the two harnesses differ in action space and available information, so guidance from the optimized harness cannot serve directly as supervision for the target one. We introduce Harness-Zero, which enables harness distillation through agent-as-harness. Guided by the optimized harness, a harnessing agent corrects student responses before execution in the target harness’s action space, turning harness guidance into training demonstrations. Fine-tuning on the resulting trajectories internalizes harness-induced behavior into the model, so the specialized harness can be removed at deployment. Our experiments spanning knowledge work, tool use, and science domains show that: (1) For frontier LLMs using the same evolved harness, agent-as-harness outperforms code-as-harness. (2) With the specialized harness removed at deployment, Harness-Zero improves the base model’s macro-average task success from 23.3% to 44.3%, even exceeding the 41.7% it reaches with that harness still attached. (3) Harness-Zero recovers harness-induced behaviors absent from the base model, with 82.3% average recovery across 28 patterns in the three domains.
13. CARE：面向视觉-语言-动作策略的经验引导式原子纠正执行
- 英文标题：CARE: Experience-Guided Atomic Corrective Execution for Vision-Language-Action Policies
- arXiv：2609.24118v1
💡 一句话解读：机器人视觉-语言-动作模型一旦偏离正常轨迹就容易失败。CARE 通过收集真实失败案例、学习各阶段出错规律，自动生成有代表性的失败场景和纠正示范，并在执行时实时监控、及时做局部调整或重做，从而提升恢复能力。
视觉-语言-动作（VLA）策略在机器人操作中表现优异，但一旦执行偏离标称轨迹便十分脆弱。我们提出 CARE（Corrective Atomic Robotic Execution，纠正性原子机器人执行），一个通过从执行中遇到的失败中学习来提升恢复能力的框架。CARE 不依赖人工设计或随机扰动来生成纠正数据，而是收集失败的执行轨迹，建模分阶段（stage-conditioned）的失败后偏差，并利用由此得到的经验分布合成具有代表性的失败状态与纠正示范。在推理阶段，CARE 将分阶段规划与基于物理的 3D 监控相结合，在保持任务进度的同时触发原子级调整或重新操作。我们进一步提出失败状态恢复基准（Failure State Recovery Benchmark, FSR-Bench），用于评估在局部偏差和结构性异常下从中间失败状态恢复的能力。在多种 VLA 主干、仿真基准和真实双臂任务上的实验显示出一致提升，仿真中平均任务成功率提升 14.5 个百分点，真实世界中提升 15.9 个百分点。代码、模型和数据见 https://github.com/xiaojunlan/care。
📄 English Abstract
Vision-Language-Action (VLA) policies achieve strong performance in robotic manipulation but remain brittle once execution deviates from nominal trajectories. We propose CARE (Corrective Atomic Robotic Execution), a framework that improves recovery by learning from failures encountered during execution. Instead of generating corrective data from manually designed or random perturbations, CARE collects failed rollouts, models stage-conditioned post-failure deviations, and uses the resulting empirical distributions to synthesize representative failure states and corrective demonstrations. At inference time, CARE combines stage-wise planning with physically grounded 3D monitoring to trigger atomic adjustments or re-operations while preserving task progress. We further introduce the Failure State Recovery Benchmark (FSR-Bench), which evaluates recovery from intermediate failure states under local deviations and structural anomalies. Experiments across multiple VLA backbones, simulation benchmarks, and real-world dual-arm tasks show consistent improvements, with average task-success gains of 14.5 points in simulation and 15.9 points in the real world. Code, models, and data are available at https://github.com/xiaojunlan/care
14. Jev-Mem：面向高效 AI 智能体的 System-One 控制型智能体记忆
- 英文标题：Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents
- arXiv：2609.23986v1
💡 一句话解读：现有 AI 智能体的记忆系统常靠大模型一步步生成来控制记忆的整理和查找，导致又慢又贵。这篇论文提出 Jev-Mem，模仿人脑“快思考/慢思考”的分工：用轻量的快思考模块负责记忆分类、检索和筛选，只在复杂推理时才动用慢思考大模型，从而让记忆更准、构建和查询更快。
智能体记忆对长程 AI 智能体正变得不可或缺，但许多现有系统依赖自回归大语言模型（LLM）来控制记忆的组织、检索与使用方式，使昂贵的生成过程处于记忆操作的关键路径上。我们提出 Jev-Mem，一种受“系统一/系统二”认知理论启发的新型智能体记忆架构：系统一负责快速、轻量的决策，系统二负责较慢、深思熟虑的推理。Jev-Mem 通过专门的系统一控制平面、结构化的多关系记忆平面和系统二推理平面，将这种分工引入智能体记忆。系统一控制器在构建阶段管理记忆类型划分与关系组织，并在检索阶段动态执行查询路由、检索预算分配、图遍历、候选打分和自适应停止；系统二仅在复杂推理与答案合成时被调用。该设计同时提升了记忆效果与系统效率：在 LoCoMo 上，Jev-Mem 取得 0.777 的 LLM-as-a-Judge 总体得分，相较最强基线相对提升 11.0%；同时将记忆构建时间降至 158 秒，比最快的竞争记忆系统快 6.6 倍，并将平均查询延迟降至 0.93 秒，减少 36.7%。
📄 English Abstract
Agentic memory is becoming essential for long-horizon AI agents, yet many existing systems rely on autoregressive LLMs to control how memories are organized, retrieved, and used, placing expensive generation on the critical path of memory operations. We introduce \textbf{\method}, a new agentic memory architecture inspired by System-One/System-Two cognition. System One captures fast, lightweight decision-making, whereas System Two performs slower, deliberative reasoning. Jev-Mem brings this division of labor to agentic memory through a dedicated System-One control plane, a structured multi-relational memory plane, and a System-Two reasoning plane. The System-One controller governs memory typing and relational organization during construction, and dynamically performs query routing, retrieval-budget allocation, graph traversal, candidate scoring, and adaptive stopping during retrieval. System Two is invoked only for complex reasoning and answer synthesis. This design improves both memory effectiveness and system efficiency: on LoCoMo Jev-Mem achieves an overall LLM-as-a-Judge score of 0.777, an 11.0% relative improvement over the strongest baseline, while reducing memory construction time to 158,s, a 6.6$\times$ speedup over the fastest competing memory system, and lowering average query latency to 0.93,s, a 36.7% reduction.
15. 为什么视频扩散模型会违背物理规律？揭示注意力机制中的缺陷
- 英文标题：Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
- arXiv：2609.23658v1
💡 一句话解读：视频扩散模型生成的视频常违背物理规律。本文首次从内部机制入手，发现是注意力机制中的旋转位置编码（RoPE）导致空间注意力过度衰减，使早期候选区域过早锁定在错误位置，抑制了合理轨迹。作者提出一种轻量级改进：在不同去噪步骤中调整RoPE频率，减少衰减，让模型探索更好的候选区域，从而生成符合物理常识的视频。
尽管视觉质量令人印象深刻，当前最先进的视频扩散模型仍常生成违背现实物理规律的内容。现有解决方案依赖外部先验或专门数据，而我们通过探索这些模型的内部机制来研究根本原因。具体而言，我们首次对文本到视频扩散模型的“运动规划”过程进行可解释性研究，揭示了运动轨迹在早期去噪阶段如何形成。基于“先形状，后细节”的发现，我们结合交叉注意力轨迹模式与因果头贡献，识别出驱动运动规划的特定注意力头子集。此外，我们的自注意力分析表明，旋转位置嵌入（RoPE）引发了过度的空间注意力衰减。这导致早期候选区域过早锁定在物理上不合理的位置，抑制了相邻帧中的合理轨迹，并触发生成失败模式。为了解决这一根本缺陷，我们提出一种轻量级架构修改，在不同去噪步骤中缩放RoPE的频率。该策略减少了过度的注意力衰减，帮助模型探索更好的候选区域以建立连贯的物理运动。最后，无需训练和基于训练的实验均证实了我们的方法在增强生成视频的物理常识方面的有效性。
📄 English Abstract
Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the ‘’motion planning’’ process of text-to-video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the ‘’first shape, then details’’ finding, we combine cross-attention trajectory patterns with causal head contributions to identify a specific subset of attention heads driving motion planning. Further, our self-attention analysis shows that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated videos.