📋 摘要
⭐ 涉及LLM agent可靠性与失败模式分析，与AI for SE、可信AI有一定关联，但非直接针对软件工程或测试。
FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environments
中文
针对开源 LLM 在多轮交互式工具使用场景（如以客户问题解决为核心的对话式 benchmark）中因决策错误级联累积而导致任务失败的问题，本文提出 Failure-Aware Meta-Agentic (FAMA) 框架。作者指出，参数规模较小、context window 有限、推理预算受限的开源 LLM 在 agentic 环境下尤其容易出现错误累积。FAMA 采用两阶段设计：第一阶段对 baseline agent 的失败轨迹进行分析，归纳出最常见的错误类型；第二阶段通过一个 orchestration 机制，在工具使用 agent 进行决策前，按需激活最小规模的专门化 agent 子集，向其注入针对相应失败模式的定向 context。在多个开源 LLM 上的实验显示，相较标准 baseline，FAMA 在不同评测模式下取得最高 27% 的性能提升。与单纯扩大模型或通用 prompting 的做法不同，本文强调通过专门化 agent 对 context 进行有针对性的策划，以缓解共性失败，是构建可靠的多轮工具使用 LLM agent 的有价值设计原则。
English abstract
Large Language Models are being increasingly deployed as the decision-making core of autonomous agents capable of effecting change in external environments. Yet, in conversational benchmarks, which simulate real-world customer-centric issue resolution scenarios, these agents frequently fail due to the cascading effects of incorrect decision-making. These challenges are particularly pronounced for open-source LLMs with smaller parameter sizes, limited context windows, and constrained inference budgets, which contribute to increased error accumulation in agentic settings. To tackle these challenges, we present the Failure-Aware Meta-Agentic (FAMA) framework. FAMA operates in two stages: first, it analyzes failure trajectories from baseline agents to identify the most prevalent errors; second, it employs an orchestration mechanism that activates a minimal subset of specialized agents tailored to address these failures by injecting a targeted context for the tool-use agent before the decision-making step. Experiments across open-source LLMs demonstrate performance gains up to 27% across evaluation modes over standard baselines. These results highlight that targeted curation of context through specialized agents to address common failures is a valuable design principle for building reliable, multi-turn tool-use LLM agents that simulate real-world conversational scenarios.
加载中…
点文件 → 加为 tab；按 Esc 关闭