The Case for "Thick Evaluations" of Cultural Representation in AI

Current AI evaluations of cultural representation are too simplistic. The paper argues that existing methods for assessing how well AI systems represent non-Western cultures rely on overly reductive standards that don't capture the real complexity of what representation means to actual communities.

Communities themselves should define what "good representation" looks like. Rather than having external evaluators decide whether AI outputs accurately represent a culture, the researchers found through South Asian workshops that people within those communities have nuanced, contextual understandings of representation that should directly shape evaluation metrics.

Representation is interpretive and contextual, not objective. The paper emphasizes that how people understand and assign meaning to cultural images varies based on individual experience, context, and perspective—meaning there's no single "correct" way an AI should represent any culture.

Co-creating evaluation metrics with communities produces better results. By working directly with South Asian communities to develop evaluation frameworks, the researchers demonstrate that measurement standards aligned with on-the-ground community experiences are more meaningful than top-down assessment approaches.

"Thick evaluations" require granular, situated, and discursive approaches. Moving beyond simple accuracy checks, responsible AI evaluation of cultural representation should involve detailed, community-engaged dialogue that acknowledges the interpretive nature of culture itself.