Abstract
Text-to-image systems increasingly rely on learned aesthetic scorers to filter training data and guide generation. Yet it remains unclear whether these scores treat demographic attributes as objective measures of quality. We audit four prominent scorers—LAION-Aesthetics, PickScore, ImageReward, and HPSv2—using pixel-level interventions on skin tone and body type across both synthetic and real images. Our central finding is that along the skin-lightness axis, the dominant effect is fidelity preference: unaltered images receive the highest scores, while perturbations in either direction are penalized (producing an inverted-U pattern). Placebo arms confirm that this penalty is not an artifact of the skin manipulation itself, as applying the same CIELAB L* shift to non-skin regions yields similar penalty magnitudes. However, the penalty is operator-dependent, and only LAION-Aesthetics consistently penalizes across all operators.
Crucially, audits conducted solely on synthetic images can be misleading. LAION-Aesthetics exhibits a strong preference for darker skin on synthetic faces, yet on 1,470 real faces this preference reverses and becomes substantially weaker, with amplification no longer significant. Across scorers, synthetic results fail to transfer: they reverse for LAION-Aesthetics and HPSv2, and attenuate for PickScore. We contribute a reproducible benchmark with artifact control and synthetic/real cross-validation, along with an auditability criterion for pixel-level causal isolation—valid for skin tone but not for body type due to deformation. Population-stratified analysis reveals that fidelity-penalty asymmetry is not robust across groups after FDR correction, except for HPSv2. Our findings demonstrate that naive synthetic audits misjudge both the direction and magnitude of bias, and that only within-image causal isolation on real data can reliably separate true demographic bias from fidelity preference.
via ArXiv CV