Prosocial Behavior Detection in Player Game Chat: From Aligning Human-AI Definitions to Efficient Annotation at Scale

Prosociality detection is fundamentally different from toxic content detection because it lacks established definitions and labeled datasets. The paper shows that detecting positive, supportive communication requires new approaches rather than adapting existing toxicity frameworks, highlighting how responsible AI systems need task-specific solutions.

Human-AI collaboration improved label quality and task clarity. The researchers used a "human-AI refinement loop" where annotators reviewed cases where GPT-4 and humans disagreed, which iteratively clarified what prosocial behavior actually means. This process is critical for emerging AI safety tasks where consensus definitions don't yet exist.

A two-stage inference system achieved 70% cost reduction without sacrificing accuracy. By using a lightweight classifier for high-confidence predictions and only escalating ~35% of ambiguous cases to the more expensive GPT-4o model, the system maintained ~0.90 precision while dramatically lowering deployment costs—demonstrating that responsible AI doesn't require unlimited computational resources.

Strategic human labeling effort matters more than scale alone. Rather than labeling massive datasets, the team used a small seed set of human examples to identify the best LLM labeling strategy, then synthesized 10,000 high-quality labels. This approach shows that thoughtful annotation design can be more efficient than brute-force data collection.

Task definition alignment between humans and AI systems is essential for responsible deployment. The paper emphasizes that when AI systems and human values diverge on what counts as "prosocial," clarifying these differences through structured collaboration prevents misaligned systems from being deployed at scale.