Mitigating Trojanized Prompt Chains in Educational LLM Use Cases: Experimental Findings and Detection Tool Design

Students can manipulate LLMs through "Trojanized prompts" that bypass safety systems: The research demonstrates that K-12 students may craft prompts designed to trick LLMs like GPT-3.5 and GPT-4 into producing unsafe or unintended content, even when the models have content moderation safeguards in place.

Multi-turn conversations create particular vulnerabilities: The study's systematic experiments using simulated K-12 queries and multi-turn dialogues revealed that safety risks increase across extended conversations, suggesting that students could gradually manipulate LLMs over multiple exchanges to circumvent protections.

Both leading AI models show exploitable weaknesses: The research exposed vulnerabilities in both GPT-3.5 and GPT-4, indicating that even state-of-the-art language models currently used in educational settings are susceptible to prompt manipulation attacks.

Detection tools can help protect educational LLM deployment: The researchers developed TrojanPromptGuard (TPG), a prototype tool designed to automatically identify and mitigate Trojanized prompts before they reach students, offering a practical safeguard for educational settings.

Proactive safety measures are needed before widespread classroom adoption: These findings suggest that educational institutions should implement detection and mitigation strategies before deploying LLMs at scale in K-12 environments, rather than assuming existing safety features are sufficient.