📋 摘要
⭐ 属于dLLM蒸馏的模型压缩工作，与SE for AI、可信AI、公平性测试等用户兴趣方向偏离较远。
Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models
中文
本文针对 diffusion large language models (dLLMs) 的跨架构知识蒸馏问题展开研究。当前 SOTA 的 dLLM 通常需要数十亿参数才能取得有竞争力的表现，而已有的 dLLM 蒸馏方法仅在同一架构内减少推理步数，未能处理 teacher 与 student 在架构、attention 机制以及 tokenizer 上均存在差异的跨架构知识迁移场景。作者提出 TIDE，首个面向 dLLM 的跨架构蒸馏框架，由三个模块化组件构成：① TIDAL 在训练进度与 diffusion timestep 两个维度上联合调节蒸馏强度，以应对 teacher 在不同噪声水平下可靠性差异的问题；② CompDemo 通过互补的 mask 切分丰富 teacher 的上下文，改善高 masking 比例下的预测质量；③ Reverse CALM 提出反向的 chunk-level likelihood matching 目标，作为跨 tokenizer 的训练信号，可获得有界梯度并实现双端噪声过滤。实验中将 8B dense 与 16B MoE 两类 teacher 经由两条异构 pipeline 蒸馏到 0.6B 的 student，在八个 benchmark 上平均较 baseline 提升 1.53 分，其中代码生成任务表现尤为突出，HumanEval 得分达到 48.78，显著高于 AR baseline 的 32.3。该工作填补了 dLLM 跨架构蒸馏的空白。
English abstract
Diffusion large language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. While existing distillation methods for dLLMs reduce inference steps within a single architecture, none address cross-architecture knowledge transfer, in which the teacher and student differ in architecture, attention mechanism, and tokenizer. We present TIDE, the first framework for cross-architecture dLLM distillation, comprising three modular components: (1) TIDAL, which jointly modulates distillation strength across training progress and diffusion timestep to account for the teacher's noise-dependent reliability; (2) CompDemo, which enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking; and (3) Reverse CALM, a cross-tokenizer objective that inverts chunk-level likelihood matching, yielding bounded gradients and dual-end noise filtering. Distilling 8B dense and 16B MoE teachers into a 0.6B student via two heterogeneous pipelines outperforms the baseline by an average of 1.53 points across eight benchmarks, yielding notable gains in code generation, where HumanEval scores reach 48.78 compared to 32.3 for the AR baseline.
加载中…
点文件 → 加为 tab；按 Esc 关闭