Irresponsible Counselors: Large Language Models and the Loneliness of Modern Humans

LLMs are increasingly used for high-stakes personal decisions despite their technical limitations: users consult these models about health, relationships, finances, education, and identity because they're accessible, inexpensive, and available 24/7—yet the paper warns this represents a fundamental mismatch between user expectations and what these systems can actually provide.

There is a critical accountability gap: while users experience LLM outputs as understanding and supportive, "there is no human agent who is straightforwardly responsible or answerable" for the advice given, creating what the authors call "advisory intimacy without a subject"—meaning users feel heard by something that cannot actually be held responsible.

Current AI ethics frameworks are insufficient: the paper argues that focusing only on developer liability, data bias, or emotional attachment to chatbots misses the core problem—the combination of technical unpredictability with LLMs' social role as trusted counselors in an age of widespread human loneliness.

The illusion of neutrality is misleading: users perceive LLMs as nonjudgmental and neutral advisors, but the paper suggests this appearance masks the fact that these systems lack genuine subjectivity, intention, or understanding—raising questions about whether "neutral" advice from a system without real comprehension can be truly trustworthy.

This creates new justice concerns: the widespread reliance on LLMs for counseling has implications for equitable access to genuine human support and raises questions about how this technology may deepen rather than solve modern loneliness.