Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment

Human disagreement is normal and informative, not a problem to hide: Current AI evaluation methods often erase disagreement by combining multiple human judgments into a single "correct" answer through voting. The thesis argues this obscures valuable information about diverse perspectives, particularly in subjective domains like content moderation and sentiment analysis where disagreement is expected and reflects genuine differences in human values.

Whose perspective gets represented matters for AI alignment: Because AI systems are trained on human feedback to align with human values, the framework emphasizes that these systems must reflect diverse human perspectives rather than averaging them away. This is especially critical for safety and fairness—if minority viewpoints are systematically excluded during evaluation, the resulting AI system may not serve or represent all communities fairly.

Current evaluation practices contribute to reproducibility problems in AI: By aggregating labels and hiding disagreement, standard evaluation methods make it harder for researchers to reproduce and understand AI system behavior. A perspective-aware framework could improve reproducibility by explicitly documenting which perspectives were represented and how they influenced model development.

Human identity and context shape AI evaluation: The thesis recognizes that disagreement often stems from people's identities, beliefs, and social environments—not from carelessness or error. Responsible AI development requires acknowledging that evaluation is inherently perspective-dependent rather than objective, and building systems that account for this reality.