Generative Image Synthesis with Diffusion Models

Problem

This line of work asked two related questions. Can diffusion models synthesize training data that is useful in practice, meaning textures diverse and faithful enough to augment real datasets? And can they generate high resolution images efficiently, without paying the full cost of diffusion in pixel space?

Approach

For texture synthesis, I trained denoising diffusion probabilistic models (DDPM) on the Amsterdam Library of Textures (ALOT) to generate artificial textures that resemble the originals while adding diversity, which makes them directly usable for augmenting texture datasets.

For latent diffusion, I implemented latent diffusion models (LDM). A pretrained autoencoder compresses images into a latent space, diffusion runs in that space instead of in pixels, and a cross-attention U-Net conditions the generation on text prompts.

Architecture & Tech

The implementation is PyTorch end to end. It covers DDPM noise schedule training on ALOT, and for the LDM work a pretrained autoencoder pair, diffusion in the latent space, and cross-attention conditioning wired into the U-Net. Evaluation used CelebA-HQ and LAION with FID and Inception Score.

Results & Impact

• The DDPM produced diverse, high fidelity artificial textures that resemble the ALOT originals, providing a practical augmentation source for texture datasets.
• The LDM implementation reproduced the core efficiency result. Diffusion in latent space delivers conditional high resolution synthesis at a fraction of the pixel space compute, quantified with FID and IS on CelebA-HQ and LAION.

What I'd do next

I would measure the downstream effect directly by training a texture classifier with and without synthetic augmentation, and I would explore fine tuning the LDM text encoder for domain specific vocabularies.