New Relic

Here's a rundown of the LLM observability tools that are genuinely popular with engineering teams right now, based on current market coverage:

Open-Source Leaders

Langfuse — Widely considered the open-source standard. Langfuse is the open source leader in this space, with over 28,000 GitHub stars and an MIT license that lets you self-host without restrictions, covering tracing with multi-turn conversation support, prompt versioning with a built-in playground, and flexible evaluation through LLM-as-judge, user feedback, or custom metrics. It has native SDKs for Python and JavaScript, plus connectors for LangChain, LlamaIndex, and 50+ other frameworks, with OpenTelemetry support for piping traces into existing observability stacks.

Arize Phoenix — A well-regarded open-source option, often cited alongside Langfuse for self-hostable observability.

MLflow — If your team already uses it for traditional ML, it's notable that MLflow is the only tool that combines open-source licensing, deep agent tracing with replay, prompt versioning, and automated evaluation in a single platform, whereas every other tool either specializes in one dimension or requires a commercial SaaS dependency.

TruLens — Good fit for research-heavy teams: it takes a qualitative-first approach to LLM observability, built around structured feedback functions that evaluate LLM responses after each call, and is particularly strong for teams using LlamaIndex and LangChain who want systematic evaluation pipelines rather than traditional tracing.

Commercial/Enterprise Platforms

LangSmith (by LangChain) — One of the most commonly used tools for teams building on the LangChain ecosystem; frequently listed among the top platforms in market comparisons.

Arize AI — A leading commercial platform for ML/LLM monitoring, often paired with its open-source Phoenix product.

Datadog — For teams already using it for infrastructure APM, Datadog has extended into LLM observability, letting AI monitoring live alongside existing infra dashboards.

Confident AI — Differentiates itself by making evaluation central: it evaluates production traces with 50+ research-backed metrics, alerts on quality and drift, and auto-curates datasets.

Comet (Opik) — Good for teams with existing ML workflows: Opik extends Comet's existing experiment tracking heritage into the GenAI space, supporting trace logging, prompt management, and evaluation scoring, with the integration story strongest for teams that want a single platform spanning classical ML model tracking and LLM observability.

Braintrust, Galileo, Fiddler AI — All show up consistently in "top 10" comparisons as commercial options focused on evaluation and production monitoring, alongside general APM vendors like Honeycomb, New Relic, and Dynatrace which have added LLM-specific tracing.

Gateway-Style Tools

Helicone — Popular for its simplicity: it sits in front of many LLM providers as a gateway, giving unified logging for prompts and completions plus cost, latency, and error visibility across vendors, with fast setup since observability is request-centric rather than deep ag

In the LLM and AI agent space, traditional application performance monitoring (APM) tools (like standard Datadog or New Relic) are often insufficient. Traditional software breaks loudly with crash logs, whereas LLM applications fail quietly—generating hallucinated outputs, getting stuck in multi-step loops, or pulling the wrong database context while returning a "200 OK" status.

Because of this, the industry has standardized on LLM observability and evaluation platforms that trace multi-step "agent" runs and automatically evaluate output quality.

1. The Big Three (The Industry Standards)

These are the most widely adopted general-purpose tools for tracing, debugging, and managing LLM prompts.

• Langfuse
  • Best for: Teams wanting a powerful, production-grade, open-source (MIT-licensed) platform.
  • The Vibe: High popularity, extremely developer-friendly, and highly customizable. It integrates well with everything (LangChain, LlamaIndex, LiteLLM, etc.).
  • Key Strengths: Excellent prompt management, SDK instrumentation, session tracking (grouping multiple LLM turns together), and cost/latency tracking. (Note: Langfuse was acquired by ClickHouse, ensuring top-tier database performance for heavy production tracing workloads).
• LangSmith
  • Best for: Teams already building with the LangChain or LangGraph ecosystems.
  • The Vibe: Extremely polished, commercial SaaS with tight, native integration into LangChain's tooling.
  • Key Strengths: Unrivaled UI for debugging complex, nested multi-agent runs (using LangGraph Studio). It seamlessly lets you turn production traces into test datasets, run offline evaluations, and manage annotation queues. It is framework-agnostic but shines brightest inside the LangChain family.
• Arize Phoenix (and Arize AI)
  • Best for: Teams prioritizing deep evaluations, vector search auditing, and OpenTelemetry-native standards.
  • The Vibe: Open-source (ELv2) local-first tool that scales to Arize’s enterprise cloud platform.
  • Key Strengths: If you are building a RAG (Retrieval-Augmented Generation) app, Phoenix is fantastic. It provides deep visibility into your vector embeddings, helps visualize chunk retrieval, and has built-in evaluators to check for retrieval relevance and hallucinations.

2. The Evaluation-First Platforms

These tools prioritize "regression testing" and LLM-as-a-judge workflows, ensuring your prompts and models improve over time without breaking existing features.

• Braintrust
  • Best for: Enterprise teams where rigorous testing, CI/CD evaluation, and prompt playground iteration are the main bottlenecks.
  • Key Strengths: Built around a highly optimized evaluation engine. It has a stellar playground where non-technical product managers can tweak prompts, run them against thousands of test cases, see the diff, and push them to production safely.
• Confident AI (DeepEval)
  • Best for: Teams that want highly structured, research-backed LLM evaluations with quality-based alerting.
  • Key Strengths: It makes evaluation the core of observability. You can score every single trace on production with 50+ built-in metrics (like toxicity, bias, and answer relevancy). If a metric drops below a threshold, it hooks directly into PagerDuty, Slack, or Teams to alert your on-call team.

3. The AI Gateways

These platforms act as a reverse proxy sitting between your application code and the LLM API providers (OpenAI, Anthropic, etc.).

• Portkey
  • Best for: Multi-provider cost tracking, caching, automatic fallbacks, and security.
  • Key Strengths: Because it sits as a gateway, it can catch rate limits and automatically route a failed Claude request to OpenAI. It provides incredibly fast latency tracking, prompt caching, and guardrail enforcement, though its trace-level agent reasoning is more high-level than LangSmith or Langfuse.
  • Alternative: Helicone is another popular gateway in this category, though its development slowed after being acquired by Mintlify.

4. Agent-Native & Emerging Players

As single-prompt LLM calls have evolved into multi-step "AI Agents" that execute code and use external tools, a new generation of agent-first tools has emerged.

• Laminar
  • Best for: Highly complex, hundreds-of-spans deep AI agent runs.
  • Key Strengths: Open-source, OpenTelemetry-native, and built from scratch specifically for multi-step agent architectures. It features massive trace compression to keep UI performance snappy when debugging loops, and features SQL querying over all stored platform data.
• Comet Opik
  • Best for: Teams looking for a fully Apache 2.0 open-source, end-to-end framework.
  • Key Strengths: Highly developer-friendly, offering self-hosted tracing, dataset curation, and automated feedback loops without vendor lock-in.
• AgentOps
  • Best for: Multi-framework agent debugging (e.g., CrewAI, AutoGen).
  • Key Strengths: Features "time-travel debugging" to replay agent sessions, trace agent state, and identify exactly where an agent wandered off course.

Summary Recommendation: Which should you choose?

Here’s a practical rundown of the LLM observability tools that people actually use, what they’re best at, and how to choose.

Quick recommendations

• Best open-source / self-hostable: Langfuse or Arize Phoenix
• Best if you live in LangChain: LangSmith
• Simplest / most lightweight: Helicone
• Best for full-stack enterprise monitoring: Datadog LLM Observability or New Relic
• Best if you already use W&B: Weights & Biases Weave
• Best if you already use MLflow / Databricks: MLflow LLM Tracking

Top tools in detail

1. Langfuse

Type: Open-source + cloud | Best for: Teams that want a full LLM engineering platform without vendor lock-in.

Langfuse is one of the most popular open-source LLM observability tools. It gives you:

• Trace/span tracking across chains, agents, and tools
• Prompt versioning and prompt management
• Evaluations and feedback collection
• Cost and latency analytics
• Self-hosting via Docker or a managed cloud option

It also integrates with OpenAI, Anthropic, LangChain, LlamaIndex, and can ingest OpenTelemetry traces.

Why people use it: Open source, actively maintained, strong community, and it’s more than just tracing — it’s a full prompt engineering platform.

2. LangSmith

Type: SaaS (by LangChain) | Best for: Anyone building production apps with LangChain.

LangSmith is the official observability platform from the LangChain team. It provides:

• Detailed traces of every step in a LangChain or agent workflow
• Dataset creation for regression testing
• Built-in evaluation suites
• Debugging tools for prompt/LLM failures
• Monitoring dashboards for production

Why people use it: If you use LangChain heavily, LangSmith is the smoothest experience because it’s deeply integrated. If you avoid LangChain, it’s less relevant.

3. Arize Phoenix

Type: Open-source | Best for: Data scientists and ML engineers who want local, panel-style experimentation.

Phoenix is Arize’s open-source LLM observability and evaluation library. It’s very popular for:

• Local tracing of LLM calls
• Embedding visualization and clustering
• Running evaluations automatically
• Experimenting before deploying to production
• Works with OpenTelemetry

Langfuse, LangSmith, Arize Phoenix, Helicone, and Comet Opik stand out as the top LLM observability tools that teams actually use in production as of 2026.

LLM observability tools track traces (prompts, responses, tool calls, agent steps), metrics (latency, cost, token usage), evaluations (LLM-as-judge, custom scorers, RAG quality, hallucination detection), prompt management, datasets/experiments, and session replays. They help debug multi-step agents, monitor production issues, control costs, and iterate on quality. There is no universal "best"—choices depend on your stack (e.g., LangChain vs. framework-agnostic), preference for self-hosting/open source, focus on evals vs. lightweight logging, and scale.

Top Recommendations

Here are the most frequently recommended and adopted options, based on GitHub adoption (stars as a proxy for community/reach), comparisons, and real-world mentions:

• Langfuse (often ranked #1 overall for open-source teams): MIT-licensed core (with some enterprise modules), ~32k GitHub stars, self-hostable via Docker/Kubernetes, strong OpenTelemetry support. It excels at end-to-end tracing, prompt management/versioning, datasets, LLM-as-judge + code evaluations, sessions, and cost analytics. Cloud has a generous free tier (~50k observations/month). Acquired by ClickHouse in early 2026 but with public commitments to open source and self-hosting. Best for: Framework-agnostic teams wanting maturity, self-host control, and a full feature set without vendor lock-in. Widely praised as the most complete open-source platform.

• LangSmith (best for LangChain/LangGraph ecosystems): Native integration with LangChain/LangGraph for near-zero-config tracing of agents, graphs, and tool calls. Strong on visual trace trees, datasets from traces, evaluations, and a prompt playground. SaaS-focused (free tier ~5k traces/month) with enterprise self-host options. Best for: Teams already deep in the LangChain stack who value seamless instrumentation and agent-specific debugging.

• Arize Phoenix (strong open-source/OTel-native pick): Source-available (Elastic License 2.0), self-hostable, OTel-native with OpenInference extensions. Focuses on tracing, evaluations, datasets, drift detection, RAG quality visualization, and embedding analysis. Good free/self-hosted tier; cloud options available. Best for: ML/engineering teams already using OpenTelemetry, those needing strong eval + diagnostics for RAG/pipelines, or avoiding pure SaaS.

• Helicone (easiest proxy-based setup): Proxy or async logging (change base URL or add a header for quick wins), excellent for cost tracking, caching, rate limiting, request logs, latency, and errors across many providers. Open-source elements and self-host options. Best for: Startups or teams wanting minimal code changes and fast time-to-value on API-level visibility and cost control.

• Comet Opik (rising for evals + tracing): Apache 2.0, fully open-source/self-hostable (~21k stars), strong on automated evaluations, tracing for agents/RAG, prompt management, and production dashboards. Backed by Comet's experiment-tracking heritage. Best for: Teams prioritizing evaluations alongside observability, or those already in the Comet ecosystem.

Other Notable Mentions

• Weights & Biases (W&B) Weave: Good if you're already using W&B for experiments; extends to LLM tracing/evals.
• Braintrust: Evals-first with strong CI/CD integration and scoring. Enterprise-leaning.
• Enterprise/full-stack options: Datadog LLM Observability or New Relic (if already in their APM ecosystem); SigNoz or PostHog for unified OTel + product analytics.
• Gateways with obs: Portkey or LiteLLM for routing + basic observability.

Quick Decision Guide

• Want open source + self-host + full features? → Start with Langfuse.
• Using LangChain/LangGraph? → LangSmith.
• OTel/ML focus or RAG diagnostics? → Arize Phoenix.
• Fastest setup, cost/caching priority? → Helicone.
• Heavy on evaluations? → Opik or Braintrust.
• Already on Datadog/W&B/etc.? → Extend with their LLM modules.

Many comparisons note that adoption (stars, mentions in communities like Reddit) favors Langfuse for broad use, LangSmith within its ecosystem, and Phoenix/Helicone for specific strengths. Test a couple with your workload—most offer free tiers or easy self-hosting. Features evolve quickly, so check current docs for integrations (Python/JS SDKs, LangChain callbacks, OTel exporters, provider proxies) and pricing at scale.

Langfuse, LangSmith, and Arize Phoenix are the top LLM observability tools that engineering teams actually use in production. Because LLM applications have unique complexities—like nested agentic loops, prompt tracking, token costs, and retrieval-augmented generation (RAG) quality—standard software monitoring tools are rarely enough on their own.

To help you find the right fit, the leading brands are organized below by their specific architectural strengths and primary use cases.

All-in-One & Developer-Favorite Frameworks

Langfuse: The most popular open-source option for teams requiring full data privacy and self-hosting capabilities.License: Fully open-source under the permissive MIT license.Core Strengths: Hierarchical trace viewing for multi-step agents, prompt versioning, and cost tracking across different models.Best For: Teams that want robust analytics but need to keep prompts and user data on their own infrastructure. Firecrawl +2 License: Fully open-source under the permissive MIT license. Core Strengths: Hierarchical trace viewing for multi-step agents, prompt versioning, and cost tracking across different models. Best For: Teams that want robust analytics but need to keep prompts and user data on their own infrastructure. LangSmith: The gold standard for debugging if your engineering team is already building in the LangChain ecosystem.Ecosystem: Developed by the creators of LangChain and LangGraph.Core Strengths: Deep, out-of-the-box tracing of agentic steps, testing loops, and automated error-clustering engines.Best For: Teams heavily reliant on LangChain who want an integrated SaaS platform that instantly hooks into their application. Maxim AI +2 Ecosystem: Developed by the creators of LangChain and LangGraph. Core Strengths: Deep, out-of-the-box tracing of agentic steps, testing loops, and automated error-clustering engines. Best For: Teams heavily reliant on LangChain who want an integrated SaaS platform that instantly hooks into their application.

Evaluation & RAG-Centric Platforms

Arize Phoenix: The top choice for teams highly focused on mathematical validation and data science experiments.Architecture: Built entirely on vendor-neutral OpenTelemetry standards.Core Strengths: Specialized in evaluating the "RAG Triad" (context relevance, groundedness, and answer relevance) to detect hallucinations.Best For: Data science-driven environments running complex retrieval systems who value open telemetry standards. LangChain +3 Architecture: Built entirely on vendor-neutral OpenTelemetry standards. Core Strengths: Specialized in evaluating the "RAG Triad" (context relevance, groundedness, and answer relevance) to detect hallucinations. Best For: Data science-driven environments running complex retrieval systems who value open telemetry standards. Braintrust: An enterprise-grade, evaluation-first platform built directly into traditional software delivery flows.Core Strengths: Exceptional CI/CD integrations that can automatically block code deployments if prompt evaluations drop below a certain threshold.Best For: Engineering-heavy teams that prioritize evaluation-driven development and strict software release controls. Pydantic +1 Core Strengths: Exceptional CI/CD integrations that can automatically block code deployments if prompt evaluations drop below a certain threshold. Best For: Engineering-heavy teams that prioritize evaluation-driven development and strict software release controls.

Lightweight AI Gateways

Helicone: A proxy-style tool designed to capture global application telemetry with minimal code modification.Architecture: Acts as a gateway between your app and the LLM providers (OpenAI, Anthropic, etc.).Core Strengths: Instant setup (usually requiring just a baseline URL or header config change), response caching, rate-limiting, and direct spend mapping.Best For: Rapid prototyping, chatbots, or teams prioritizing swift setup and cost controls over deep internal application tracing. Reddit·r/LangChain +3 Architecture: Acts as a gateway between your app and the LLM providers (OpenAI, Anthropic, etc.). Core Strengths: Instant setup (usually requiring just a baseline URL or header config change), response caching, rate-limiting, and direct spend mapping. Best For: Rapid prototyping, chatbots, or teams prioritizing swift setup and cost controls over deep internal application tracing. Portkey: A production-hardened AI gateway combining core logging with heavy traffic management.Core Strengths: Multi-provider routing, fallback model triggers, automated request retries, and unified guardrails.Best For: Large production apps that need resilient fallback infrastructure alongside their monitoring analytics. DEV Community +2 Core Strengths: Multi-provider routing, fallback model triggers, automated request retries, and unified guardrails. Best For: Large production apps that need resilient fallback infrastructure alongside their monitoring analytics.

Legacy APM Add-Ons

If your company already uses premium, full-scale infrastructure monitoring tools, look into Datadog LLM Observability or New Relic AI Monitoring.

The Tradeoff: They allow you to look at your LLM token usage right next to your standard server infrastructure logs. However, traditional APMs usually lack the hyper-specific, multi-turn agent simulation tools, playground sandboxes, and offline evaluation datasets found in AI-native platforms like Langfuse or LangSmith.

To narrow this down, what does your current technical stack look like? Knowing if you use frameworks like LangChain/LlamaIndex or if you have strict data privacy requirements will help determine the exact best fit.