DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher

DUET addresses a key challenge in trustworthy AI: The method tackles "unlearning"—removing unwanted knowledge from AI models without rebuilding them from scratch—which the authors identify as "indispensable towards trustworthy AI."

Existing unlearning approaches have significant trade-offs: Previous methods either require heavy computation and risk "catastrophic forgetting" (losing other important knowledge), or are lightweight but vulnerable to attacks where users can bypass the unlearning through prompt manipulation or reverse engineering.

DUET combines the best of both approaches: By using a "distillation-based" technique where a student model learns from a teacher model, DUET achieves both effective removal of undesirable knowledge and preservation of the model's general capabilities—addressing limitations of prior methods.

Dramatic efficiency gains: The method is "orders of magnitude more data-efficient than state-of-the-art unlearning methods," meaning organizations can implement responsible AI practices without prohibitive computational costs.

Improved robustness and performance: Evaluations show DUET achieves "higher performance in both forgetting and utility preservation," suggesting it successfully removes problematic knowledge while maintaining the model's usefulness for legitimate tasks.