Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazil

LLMs perform comparably to human test-takers on standardized exams: The study found that GPT-3.5, GPT-4, and MariTalk showed no significant biases when compared to Brazilian students on the ENEM's multiple-choice sections, with model performance differences primarily reflecting overall human accuracy levels rather than systematic bias.

Socioeconomic status disparities in humans don't translate to LLM biases in the same way: By analyzing answer distributions across different socioeconomic groups, the researchers found that LLMs did not replicate or amplify the performance gaps that exist between higher and lower SES human test-takers, suggesting these models may not inherit certain societal inequalities in this testing context.

Essay quality differs between humans and LLMs in measurable ways: When examining the writing component of ENEM, the study identified key differences between human and LLM-generated essays, including vocabulary choices, indicating that while LLMs can produce test-appropriate responses, their writing style remains distinguishable from human work.

Portuguese-language models warrant specific evaluation: The inclusion of MariTalk, a model trained on Portuguese data, alongside English-language models (GPT-3.5 and GPT-4) demonstrates the importance of testing LLMs in their specific language and cultural contexts rather than assuming findings from English-language models generalize globally.

Public datasets enable transparent AI bias assessment: The study leverages Brazil's transparency policies around ENEM data to create a rigorous framework for comparing AI behavior against real human populations, suggesting that similar public datasets could support responsible AI evaluation in other domains and countries.