AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data

CVPR 2025

Zengqun Zhao Ziquan Liu Yu Cao Shaogang Gong Ioannis Patras

Queen Mary University of London

Paper Code

Overview

Modern text-to-image models can produce highly realistic images, yet their potential to improve algorithmic fairness through targeted data expansion remains underexplored. This work asks: can synthetic data help mitigate bias in models trained on imbalanced real-world datasets?

We conduct a comprehensive empirical study of whether fine-tuning with high-quality, balanced synthetic data can correct such biases. We identify two central challenges:

Data quality and diversity. Ambiguous prompts and generator misrepresentation can produce low-quality or insufficiently diverse samples.

Safe adaptation. Domain and bias shifts between real and synthetic data mean that naive fine-tuning can improve fairness at the cost of predictive utility.

Method

AIM-Fair combines contextual data generation with parameter-selective adaptation to transfer fairness from balanced synthetic data while retaining the utility learned from real pretraining data.

01 · CSDG

Contextual Synthetic Data Generation

Uses GPT-4-generated prompts to produce diverse contextual images for balanced fine-tuning.

02 · SMG

Selective Mask Generation

Creates a selection mask that determines which model parameters are updated.

03 · SFT

Selective Fine-Tuning

Updates selected parameters to improve fairness while preserving pretrained model utility.

AIM-Fair framework showing contextual synthetic data generation, selective mask generation, and selective fine-tuning
Overview of AIM-Fair: contextual prompts generate balanced synthetic data, parameter-wise gradient differences identify a selective mask, and fine-tuning updates only the selected parameters.

Results

Comparison with prior methods on CelebA across different target and protected attributes.

AIM-Fair comparison results across target and protected attributes on CelebA

Comparison with prior methods on CelebA (T=Smiling, P=Male) across different training-set sizes.

AIM-Fair comparison results across different CelebA training-set sizes

Comparison of different fine-tuning strategies on CelebA and UTKFace.

Comparison of fine-tuning strategies on CelebA and UTKFace

Results on CelebA (T=Smiling, P=Male) across different prompt types and prompt counts.

AIM-Fair results across prompt types and prompt counts on CelebA

Visualizations

T-SNE comparison of learned CelebA representations before and after fairness fine-tuning
t-SNE visualizations of learned representations on CelebA (T=Smiling, P=Male).
Contextual synthetic CelebA images generated for Smiling and Young attributes
Generated contextual images for CelebA with target attribute Smiling and protected attribute Young.
Contextual synthetic UTKFace images generated for Female and White attributes
Generated contextual images for UTKFace with target attribute Female and protected attribute White.

BibTeX

@inproceedings{zhao2025aim,
                title={{AIM-Fair}: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data},
                author={Zhao, Zengqun and Liu, Ziquan and Cao, Yu and Gong, Shaogang and Patras, Ioannis},
                booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
                pages={28748--28758},
                year={2025}
}