Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation

Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait, Ansar Khangeldin, Karen Sanchez, Tong Zhang, Bernard Ghanem

Abstract

Proposes matching each text-to-image evaluation skill (e.g. counting, spatial relations, attribute binding) to an annotation strategy suited to its own characteristics, rather than scoring every skill the same way — improving inter-annotator agreement and giving more stable model comparisons than uniform Likert or binary-QA baselines.