Stable Diffusion image forensics

Problem

Modern diffusion models can generate faces that are difficult to distinguish from authentic photographs. Developed with Malek Mohamed Bennabi at Université Paris Cité, this project investigated whether visual and frequency-domain traces could reliably separate real faces from images produced by Stable Diffusion—and whether those detectors would transfer to unseen generation methods.

Approach

Three detection strategies were designed and compared. The first fine-tuned VGG16 and DenseNet classifiers directly on RGB images. The second converted grayscale images into log-magnitude Fourier spectra before classification, targeting frequency artifacts introduced by diffusion and denoising. The third trained an autoencoder on 60,000 CelebA faces, reused its 2,048-dimensional latent representation for binary classification, and applied MTCNN face extraction to focus the analysis on relevant regions.

Training primarily used real IMDB-WIKI faces and Stable Diffusion Inpainting samples from the 120,000-image DeepFakeFace dataset. Generalization was then evaluated on unseen Text2Img, InsightFace, and StyleGAN data. The reported protocol used balanced 10,000-image evaluation sets and tracked accuracy, precision, recall, F1 score, and ROC AUC.

Results

The best direct model, DenseNet, reached 97.75% accuracy and 99.65% AUC on Inpainting images and generalized well to unseen Stable Diffusion Text2Img samples, with 95.10% accuracy and 99.22% AUC. Its performance dropped close to chance on InsightFace and StyleGAN samples, exposing a strong dependency on the image-generation family.

Fourier-spectrum classification achieved 92.05% accuracy and 99.12% AUC on Inpainting. Although weaker on Text2Img, it transferred better than the direct CNN to InsightFace and StyleGAN, reaching 71.30% and 65.67% accuracy, respectively. The autoencoder approach was substantially less effective, showing that compact reconstruction features were not sufficiently discriminative for this task.

Takeaway

The experiments showed that high in-domain detection scores do not imply robust deepfake detection. Spatial CNNs captured diffusion-specific cues effectively, while frequency features offered a different and sometimes more transferable signal. The most promising next step is therefore a hybrid spatial-frequency model trained on a broader mix of generators, with generator-disjoint evaluation built into the protocol.