Large-scale text-to-image diffusion models pose safety and compliance risks due to their ability to generate harmful or undesirable content. Machine unlearning seeks to remove specific concepts from pretrained models while preserving overall generative quality. Low-Rank Adaptation ({LoRA}) enables parameter-efficient targeted forgetting, but naive {LoRA}-based unlearning often induces a distributional shift, degrading fidelity and destabilizing sampling. In this paper, we propose UnGuide, a {LoRA}-based unlearning framework with stability-aware adaptive guidance. Our method dynamically modulates classifier-free guidance ({CFG}) based on the variance of the early denoising trajectory. By estimating discrepancies between the base model’s and the {LoRA}-adapted model’s noise predictions during the initial diffusion steps, UnGuide interpolates between the frozen and adapted models for each prompt. High-variance trajectories receive stronger guidance from the adapted model to enforce forgetting, while low-variance trajectories rely on the base model to preserve fidelity. This mechanism enables controlled concept suppression without modifying prompt embeddings or using external segmentation. Experiments on object erasure and explicit content removal demonstrate improved forgetting efficacy and superior fidelity retention compared to existing {LoRA}-based approaches.
Authors: Alicja Polowczyk, Agnieszka Polowczyk, Dawid Malarz, Artur Kasymov, Jacek Tabor, Marcin Mazur, Przemysław Spurek