Generative AI in Writing Research Papers: A New Type of Algorithmic Bias and Uncertainty in Scholarly Work

Generative AI introduces "context-induced algorithmic bias" in research writing: Unlike traditional AI biases that stem from limited training data, LLMs like ChatGPT are trained on massive, opaque datasets where biases are difficult to identify and address. The paper warns that these hidden biases can become embedded in scholarly work when researchers use these tools to compose manuscripts.

AI hallucinations and unpredictability undermine research reproducibility: The paper identifies that generative AI tools are susceptible to "hallucinations" (generating false information) and that subtle variations in how researchers prompt these tools introduce uncertainty and irreproducibility into the research process—a fundamental requirement for scientific validity.

Researchers may unintentionally trigger harmful outputs through normal use: The paper notes that researchers can inadvertently perform "adversarial attacks" (like red teaming prompts) simply through normal interaction with chatbots, potentially generating misleading or harmful content that then gets reinforced when incorporated into published research.

AI interpretability efforts lag dangerously behind development: The paper emphasizes that our ability to understand why generative AI produces specific outputs has not kept pace with the rapid deployment of these tools in research, making it difficult for scholars to identify and correct problems before publication.

The growing use of generative AI in manuscript writing creates systemic risks: As more researchers use these tools to compose papers, biased or hallucinated content becomes increasingly embedded in the scholarly record with limited mechanisms to detect or prevent it.