LLM Unlearning via Loss Adjustment with Only Forget Data

Unlearning can work without retaining data: The paper demonstrates that LLMs can successfully "unlearn" problematic information (like privacy leaks, biased content, or unsafe material) without needing to keep copies of data the model should remember, addressing a major practical barrier to implementing unlearning in real systems.

The method balances forgetting with preserving usefulness: FLAT achieves the critical balance between making models forget targeted information while maintaining their overall performance on other tasks—a challenge previous approaches struggled with because similar queries often produce similar responses across different topics.

Template responses guide what models should do instead: Rather than just telling models what not to say, FLAT uses template answers to guide what models should say when encountering forget data, providing a more stable and practical approach than previous methods that directly pushed models away from unwanted responses.

Simpler approach reduces computational complexity: By eliminating the need for a reference model or separate retain dataset, FLAT simplifies the unlearning process, making it more feasible for organizations to implement responsible AI practices like removing biased or private information from deployed models.

Addresses evolving regulatory requirements: The paper frames unlearning as essential for compliance with regulations requiring AI systems to remove or correct problematic content, making this work relevant to organizations navigating privacy laws and AI governance frameworks.