LLMography: Transforming Human-AI Conversations into Traceability, Oversight, and Auditability Indicators

Conversation history matters more than final output alone: Rather than just detecting whether AI generated something, LLMography examines the entire interaction process to reveal how much humans directed the work, where AI contributed, and how outputs were validated—providing a more complete picture of AI-assisted work than current detection methods.

A new transparency standard is needed: The framework proposes "LLMography" labels (analogous to bibliographies) that document human-AI collaboration as a structured record, enabling educators, employers, and auditors to understand the provenance and human contribution behind AI-assisted outputs rather than treating them as black boxes.

Most human-AI interactions show meaningful human control: In the preliminary study of 19 engineering student interactions, average Human Direction Scores were 86.8/100, suggesting that when conversation histories are examined, humans typically maintain substantial oversight and direction of AI-assisted work—contradicting concerns that AI simply takes over tasks.

Auditability and traceability remain significant challenges: While human direction was high, Auditability Scores averaged only 72.8/100 and Final Output Traceability 77.1/100, indicating that even with conversation records available, it can be difficult to clearly trace how specific outputs were produced and verify their reliability.

Privacy risks emerge from detailed interaction records: Documenting complete Human-AI conversations for oversight purposes creates new privacy concerns that must be carefully managed, as these traces contain sensitive information about how individuals work and think.