AI2030 Responsible AI Tools Marketplace

Key Takeaways

Text-to-image AI models can be manipulated to produce biased outputs: The research demonstrates that these popular generative models exhibit "unpredictable behaviors and vulnerabilities" that can be exploited to distort how they represent different groups or concepts, raising concerns about their reliability in real-world applications.
Current models lack sufficient diversity in their visual outputs: The paper identifies that text-to-image generators often produce a narrow "breadth of visual representations" for learned concepts, meaning they may generate limited variations when asked to create images of certain groups or ideas—potentially reinforcing stereotypes rather than reflecting real-world diversity.
Small changes to prompts can reveal hidden biases: By testing how models respond to minor alterations in text inputs, researchers can identify when and how these systems produce unfair or unreliable results, providing a method to detect which models have embedded biases before they're widely deployed.
Fairness requires examining what happens when concepts are removed from prompts: The research introduces a new fairness metric that tests whether removing certain words from instructions still causes the model to generate biased outputs, revealing whether biases are deeply embedded in the model's training rather than simply responding to user input.
Transparency tools are now available for accountability: The authors have released public code enabling researchers and developers to evaluate their own text-to-image models for these fairness and reliability issues, supporting efforts to identify and potentially remediate biased systems before deployment.

Abstract

The rapid proliferation of multimodal generative models has sparked critical discussions on their reliability, fairness and potential for misuse. While text-to-image models excel at producing high-fidelity, user-guided content, they often exhibit unpredictable behaviors and vulnerabilities that can be exploited to manipulate class or concept representations. To address this, we propose an evaluation framework to assess model reliability by analyzing responses to global and local perturbations in the embedding space, enabling the identification of inputs that trigger unreliable or biased behavior. Beyond social implications, fairness and diversity are fundamental to defining robust and trustworthy model behavior. Our approach offers deeper insights into these essential aspects by evaluating: (i) generative diversity, measuring the breadth of visual representations for learned concepts, and (ii) generative fairness, which examines the impact that removing concepts from input prompts has on control, under a low guidance setup. Beyond these evaluations, our method lays the groundwork for detecting unreliable, bias-injected models and tracing the provenance of embedded biases. Our code is publicly available at https://github.com/JJ-Vice/T2I_Fairness_Diversity_Reliability. Keywords: Fairness, Reliability, AI Ethics, Bias, Text-to-Image Models

This research is supported by the NISDRG project #20100007, funded by the Australian Government

Key Takeaways

Text-to-image AI models can be manipulated to produce biased outputs: The research demonstrates that these popular generative models exhibit "unpredictable behaviors and vulnerabilities" that can be exploited to distort how they represent different groups or concepts, raising concerns about their reliability in real-world applications.
Current models lack sufficient diversity in their visual outputs: The paper identifies that text-to-image generators often produce a narrow "breadth of visual representations" for learned concepts, meaning they may generate limited variations when asked to create images of certain groups or ideas—potentially reinforcing stereotypes rather than reflecting real-world diversity.
Small changes to prompts can reveal hidden biases: By testing how models respond to minor alterations in text inputs, researchers can identify when and how these systems produce unfair or unreliable results, providing a method to detect which models have embedded biases before they're widely deployed.
Fairness requires examining what happens when concepts are removed from prompts: The research introduces a new fairness metric that tests whether removing certain words from instructions still causes the model to generate biased outputs, revealing whether biases are deeply embedded in the model's training rather than simply responding to user input.
Transparency tools are now available for accountability: The authors have released public code enabling researchers and developers to evaluate their own text-to-image models for these fairness and reliability issues, supporting efforts to identify and potentially remediate biased systems before deployment.

Abstract

This research is supported by the NISDRG project #20100007, funded by the Australian Government

On the Fairness, Diversity and Reliability of Text-to-Image Generative Models

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

On the Fairness, Diversity and Reliability of Text-to-Image Generative Models

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links