AI2030 Responsible AI Tools Marketplace

Key Takeaways

Character training alone is risky at scale: The paper finds that relying solely on character shaping (like RLHF) without rule-based safeguards becomes increasingly problematic as AI systems are deployed more widely, because shaped behaviors can degrade or fail in novel situations—a phenomenon the authors call "character fragility."
The optimal safety strategy depends heavily on how fragile trained behavior is: The research identifies baseline character fragility rate as "the dominant parameter," shifting the optimal balance between rules and character shaping by up to 0.50 on their scale—far more influential than filter quality or other factors. This suggests organizations should prioritize understanding and measuring how robust their AI's trained values actually are.
A hybrid approach is generally safer than pure strategies: Across most scenarios tested, the optimal safety design uses both character shaping and rule enforcement together rather than relying on either approach alone, suggesting that layered defenses remain important even as AI systems scale up.
The right balance shifts gradually with deployment scale: As AI systems are deployed to more users or contexts, the optimal allocation should shift modestly toward character shaping (between 1-21% depending on conditions), but this shift is much smaller than the influence of how fragile the trained behavior proves to be in practice.
Tail risks and expected harms converge at large scale: The paper's analysis of worst-case scenarios (CVaR) and average-case harms reach similar conclusions at large deployment scales, suggesting that both typical and catastrophic failure modes point toward similar safety design recommendations.

Abstract

Artificial Intelligence (AI) safety systems combine character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI), which modifies behavioral distributions at training time, with rule enforcement (e.g., output filters, safety classifiers), which blocks harmful outputs at inference time, yet little formal analysis exists on how their optimal balance should change as deployment scales increase. We introduce a stylized comparative-statics model that parameterizes safety design as a resource allocation alpha in [0,1] between these two approaches, incorporating scale-dependent filter degradation, common-mode failures, and character fragility -- the risk that shaped behavior degrades or collapses under novel conditions. Under a multiplicative Pareto damage model, we derive closed-form expected harm and supplement it with tail-risk (CVaR) analysis via Monte Carlo simulation. Across three scenarios (optimistic, moderate, pessimistic), the optimal alpha* is interior or at the rules-only boundary and shifts weakly toward character shaping as deployment scale T grows, from negligible (Delta alpha* = +0.01) to pronounced (Delta alpha* = +0.21) depending on scenario. The dominant parameter is the baseline character fragility rate p^(0)_frag, which shifts alpha* by 0.50 across its range -- far exceeding the effect of tail severity, filter quality, or common-mode failure probability. CVaR and expected-harm optima converge at large T. These results suggest that safety architecture decisions depend less on deployment scale per se than on the reliability of character shaping under distributional shift.

Accepted at AIES 2026 (9th AAAI/ACM Conference on AI, Ethics, and Society). 10 pages, 6 figures, 4 tables

Key Takeaways

Character training alone is risky at scale: The paper finds that relying solely on character shaping (like RLHF) without rule-based safeguards becomes increasingly problematic as AI systems are deployed more widely, because shaped behaviors can degrade or fail in novel situations—a phenomenon the authors call "character fragility."
The optimal safety strategy depends heavily on how fragile trained behavior is: The research identifies baseline character fragility rate as "the dominant parameter," shifting the optimal balance between rules and character shaping by up to 0.50 on their scale—far more influential than filter quality or other factors. This suggests organizations should prioritize understanding and measuring how robust their AI's trained values actually are.
A hybrid approach is generally safer than pure strategies: Across most scenarios tested, the optimal safety design uses both character shaping and rule enforcement together rather than relying on either approach alone, suggesting that layered defenses remain important even as AI systems scale up.
The right balance shifts gradually with deployment scale: As AI systems are deployed to more users or contexts, the optimal allocation should shift modestly toward character shaping (between 1-21% depending on conditions), but this shift is much smaller than the influence of how fragile the trained behavior proves to be in practice.
Tail risks and expected harms converge at large scale: The paper's analysis of worst-case scenarios (CVaR) and average-case harms reach similar conclusions at large deployment scales, suggesting that both typical and catastrophic failure modes point toward similar safety design recommendations.

Abstract

Accepted at AIES 2026 (9th AAAI/ACM Conference on AI, Ethics, and Society). 10 pages, 6 figures, 4 tables

Rules or Character? Scaling Laws for AI Safety Design

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links

Rules or Character? Scaling Laws for AI Safety Design

Key Takeaways

Abstract

Paper Info

Responsible AI Topics

Links