Arize Phoenix

Overview

Arize Phoenix is an open-source AI observability and evaluation platform designed to help developers trace, evaluate, and improve AI agents. It can be run locally or self-hosted within your own environment, providing visibility into AI application behavior without requiring external infrastructure. Phoenix enables teams to understand how AI agents make decisions and identify where systems fail or produce suboptimal results.

Key Features

Tracing: Provides complete visibility into agent execution by recording every step—including prompts, retrievals, tool calls, and outputs. This allows developers to investigate issues and understand the full decision-making path of their AI systems.

Evaluation: Enables measurement and improvement of agent quality through customizable evaluation functions that score outputs and catch issues before they reach users. Supports LLM-as-judge evaluation approaches.

Clustering & Drift Detection: Includes embedded clustering capabilities for identifying patterns in data and detecting drift in model behavior over time.

Annotations & Feedback: Supports human review and LLM-based annotations to mark successful outcomes and flag failures, enabling systematic quality improvement.

Experimentation: Allows teams to create datasets from traces, run controlled experiments with different configurations, and measure improvements with evidence-based results.

Agent Integrations: Provides native integrations with coding agents and developer tools, enabling automation and seamless workflow integration.

OpenInference Specification: Uses telemetry data standards to understand LLM execution and surrounding application context.

Use Cases

• LLM Debugging: Visualize complex LLM decision-making to identify hallucinations, incorrect generalizations, and failure points
• Retrieval Optimization: Troubleshoot problems related to retrieval systems and tool execution within LLM workflows