How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses

Essay length and style shifted noticeably after ChatGPT's release: The study found that student essays submitted to a university-level MOOC on AI ethics showed significant changes in both length and writing style following ChatGPT's launch in late 2022, suggesting LLMs are already influencing how students compose written work.

Students are discussing AI and LLMs more, but core topics remain stable: While the prevalence of key words related to AI and large language models increased in post-ChatGPT essays, the underlying themes and topics students engaged with did not fundamentally change, indicating that LLM availability may be affecting vocabulary choices rather than reshaping educational content.

The concern about academic dishonesty appears partially validated by observable changes: The measurable shifts in essay characteristics (length and style) align with worries expressed in academic communities about potential cheating, though the study does not directly measure dishonesty rates, suggesting educators should monitor these patterns as potential indicators.

Real-world data from online education provides concrete evidence of LLM impact: By analyzing actual student submissions before and after ChatGPT's release rather than relying on speculation, this research offers empirical grounding for the ongoing debate about how LLMs are affecting education, moving beyond theoretical concerns to documented behavioral changes.