How do you feel? Measuring User-Perceived Value for Rejecting Machine Decisions in Hate Speech Detection

User values matter more than accuracy alone: The research found that optimizing hate speech detection systems based on what users actually care about—rather than just maximizing accuracy—can deliver better overall outcomes. This suggests platforms should directly measure what their communities value when moderating harmful content, not just chase higher accuracy metrics.

Different mistakes have different costs: The study revealed that users perceive incorrect machine decisions differently depending on the type of error. For example, falsely labeling something as hate speech may carry different user-perceived costs than missing actual hate speech. Current systems often treat all errors equally, which may not align with what communities actually need.

Sometimes saying "I don't know" is valuable: The research shows that when AI systems reject uncertain decisions and pass them to humans instead of guessing, users perceive this as valuable. This challenges the assumption that AI systems should always make a decision, and suggests that knowing when to defer to human judgment is an important capability.

Magnitude Estimation offers a better measurement tool: Rather than using traditional rating scales (like 1-5 stars), the study found that an unbounded scale called Magnitude Estimation more reliably captured how users actually felt about machine decisions in hate speech detection. This methodological finding could improve how platforms measure user satisfaction with moderation systems.

Collaborative systems need human input on values: The paper demonstrates that building effective human-AI collaboration requires understanding user perspectives upfront, not just optimizing technical performance. Platforms should conduct community input studies to understand what their users value before designing moderation systems.