Evaluation of large language models using an Indian language LGBTI+ lexicon

LLMs fail to detect hate speech targeting LGBTI+ communities in Indian languages: The study found that three tested large language models were unable to identify hateful content directed at LGBTI+ individuals, suggesting these systems may inadvertently amplify discrimination against marginalized groups.

Standard benchmarks like MMLU don't measure responsible AI behavior: Traditional performance metrics focus on task completion rather than ethical considerations, leaving significant gaps in understanding how LLMs handle sensitive social contexts like LGBTI+ issues.

Machine translation is an unreliable shortcut for non-English language evaluation: The researchers discovered that simply translating English benchmarks into other languages doesn't adequately test language understanding, meaning LLMs may behave differently in non-English contexts than current evaluation methods suggest.

Domain-specific lexicons are necessary for responsible AI assessment: Creating specialized dictionaries of terminology relevant to particular communities (like LGBTI+ terms in Indian languages) provides a more targeted way to evaluate whether AI systems handle sensitive topics appropriately.

A replicable methodology for testing cultural context: The paper offers a four-step evaluation framework that other researchers can apply to different languages and communities, potentially helping identify and address bias across diverse populations and social groups.