Disaggregated Health Data in LLMs: Evaluating Data Equity in the Context of Asian American Representation

LLMs often fail to provide disaggregated health data for Asian American sub-groups: The study found that popular models like ChatGPT and Claude struggle to retrieve specific health information for distinct communities (Korean, Chinese, etc.) rather than treating Asian Americans as a monolithic group, which can mask important health disparities.

Data aggregation in LLMs perpetuates existing health inequities: By lumping diverse populations together, these models reproduce the same problematic practices that health researchers have worked to overcome through disaggregation, potentially leading users to miss critical health information relevant to specific communities.

LLMs are increasingly used as primary information sources without adequate equity safeguards: As people turn to ChatGPT and Claude instead of traditional search engines for health information, the models' inability to provide accurate, disaggregated data raises serious concerns about health equity and informed decision-making across different demographic groups.

Current LLM evaluation methods are insufficient for assessing representation equity: The paper demonstrates that specialized statistical and machine learning tools are needed to properly evaluate whether LLMs serve diverse populations fairly—standard performance metrics alone don't capture these equity gaps.

Responsible AI requires attention to how models handle complex, intersectional health disparities: The focus on Asian American sub-ethnic groups reveals that ensuring equitable AI systems means going beyond simple demographic categories to address how models represent nuanced health differences within and across communities.