📋 摘要
⭐ 聚焦扩散模型可控生成插件框架，与SE for AI、ML测试、公平性测试等方向几乎无交集。
Diffusion Templates: A Unified Plugin Framework for Controllable Diffusion
中文
针对当前可控 diffusion 方法多以孤立、绑定特定 backbone 的形式开发，存在训练流程、参数格式和运行时接口互不兼容，难以跨任务复用基础设施、跨 backbone 迁移能力或在单一生成流水线中组合多种控制等问题，本文提出 Diffusion Templates，一种统一且开放的插件式框架，将基础模型推理与可控能力注入解耦。该框架由三部分构成：Template models 将任意任务特定输入映射为中间能力表征；Template cache 作为标准化的能力注入接口；Template pipeline 负责加载、合并并将一个或多个 Template cache 注入基础 diffusion 运行时。由于该接口在系统层面而非特定控制架构层面定义，KV-Cache 与 LoRA 等异构能力载体可在同一抽象下得到支持。基于此设计，作者构建了覆盖结构控制、亮度调节、色彩调节、图像编辑、超分辨率、锐度增强、美学对齐、内容参考、局部 inpainting 与年龄控制的多样化 model zoo。案例研究表明，Diffusion Templates 能够统一多类可控生成任务，并在快速演进的 diffusion backbone 之上保持模块化、可组合性与实用可扩展性。相关代码、模型与数据集将全部开源。
English abstract
Controllable diffusion methods have substantially expanded the practical utility of diffusion models, but they are typically developed as isolated, backbone-specific systems with incompatible training pipelines, parameter formats, and runtime hooks. This fragmentation makes it difficult to reuse infrastructure across tasks, transfer capabilities across backbones, or compose multiple controls within a single generation pipeline. We present Diffusion Templates, a unified and open plugin framework that decouples base-model inference from controllable capability injection. The framework is organized around three components: Template models that map arbitrary task-specific inputs to an intermediate capability representation, a Template cache that functions as a standardized interface for capability injection, and a Template pipeline that loads, merges, and injects one or more Template caches into the base diffusion runtime. Because the interface is defined at the systems level rather than tied to a specific control architecture, heterogeneous capability carriers such as KV-Cache and LoRA can be supported under the same abstraction. Based on this design, we build a diverse model zoo spanning structural control, brightness adjustment, color adjustment, image editing, super-resolution, sharpness enhancement, aesthetic alignment, content reference, local inpainting, and age control. These case studies show that Diffusion Templates can unify a broad range of controllable generation tasks while preserving modularity, composability, and practical extensibility across rapidly evolving diffusion backbones. All resources will be open sourced, including code, models, and datasets.
加载中…
点文件 → 加为 tab；按 Esc 关闭