AI2030 Responsible AI Tools Marketplace

Key Takeaways

VLMs can adapt outputs based on cultural cues, but inconsistently: The study found that Vision-Language Models do incorporate culturally-specific vocabulary (names, family terms, geographic references) when given cultural identity signals, demonstrating some capability to recognize and respond to cultural context during story generation tasks.
Performance varies dramatically between different AI models: Different VLM architectures showed significantly different levels of cultural competence, and some models even exhibited "inverse cultural alignment"—meaning they sometimes produced outputs misaligned with the cultural identity cues provided—highlighting that cultural competence is not a universal feature across AI systems.
Automated evaluation metrics may mask real problems: The researchers found that automated metrics used to evaluate these models contradicted human assessments, suggesting that standard technical benchmarks may not accurately capture whether AI systems are truly culturally competent, which has important implications for how we assess responsible AI deployment.
Visual and cultural understanding remain disconnected: While the models could generate culturally-distinct text outputs, their visual-cultural understanding was limited, with only 28.7% recall for within-nationality visual-semantic similarity compared to 0.2% cross-nationality—indicating that VLMs struggle to genuinely understand cultural context from images.
Systematic evaluation of cultural competence is now possible but urgent: This is the first comprehensive framework for testing how generative VLMs handle cultural identity across both text and images, establishing a foundation for assessing cultural competence before these systems are deployed globally.

Abstract

As Vision-Language Models (VLMs) achieve widespread deployment across diverse cultural contexts, ensuring their cultural competence becomes critical for responsible AI systems. While prior work has evaluated cultural awareness in text-only models and VLM object recognition tasks, no research has systematically assessed how VLMs adapt outputs when cultural identity cues are embedded in both textual prompts and visual inputs during generative tasks. We present the first comprehensive evaluation of VLM cultural competence through multimodal story generation, developing a novel multimodal framework that perturbs cultural identity and evaluates 5 contemporary VLMs on a downstream task: story generation. Our analysis reveals significant cultural adaptation capabilities, with rich culturally-specific vocabulary spanning names, familial terms, and geographic markers. However, we uncover concerning limitations: cultural competence varies dramatically across architectures, some models exhibit inverse cultural alignment, and automated metrics show architectural bias contradicting human assessments. Cross-modal evaluation shows that culturally distinct outputs are indeed detectable through visual-semantic similarity (28.7% within-nationality vs. 0.2% cross-nationality recall), yet visual-cultural understanding remains limited. In essence, we establish the promise and challenges of cultural competence in multimodal AI. We publicly release our codebase and data: https://github.com/ArkaMukherjee0/mmCultural

Accepted at ASI @ ICCV 2025

Key Takeaways

VLMs can adapt outputs based on cultural cues, but inconsistently: The study found that Vision-Language Models do incorporate culturally-specific vocabulary (names, family terms, geographic references) when given cultural identity signals, demonstrating some capability to recognize and respond to cultural context during story generation tasks.
Performance varies dramatically between different AI models: Different VLM architectures showed significantly different levels of cultural competence, and some models even exhibited "inverse cultural alignment"—meaning they sometimes produced outputs misaligned with the cultural identity cues provided—highlighting that cultural competence is not a universal feature across AI systems.
Automated evaluation metrics may mask real problems: The researchers found that automated metrics used to evaluate these models contradicted human assessments, suggesting that standard technical benchmarks may not accurately capture whether AI systems are truly culturally competent, which has important implications for how we assess responsible AI deployment.
Visual and cultural understanding remain disconnected: While the models could generate culturally-distinct text outputs, their visual-cultural understanding was limited, with only 28.7% recall for within-nationality visual-semantic similarity compared to 0.2% cross-nationality—indicating that VLMs struggle to genuinely understand cultural context from images.
Systematic evaluation of cultural competence is now possible but urgent: This is the first comprehensive framework for testing how generative VLMs handle cultural identity across both text and images, establishing a foundation for assessing cultural competence before these systems are deployed globally.

Abstract

Accepted at ASI @ ICCV 2025

Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links