> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt # The best open source LLM observability tools, compared Canonical URL: https://posthog.com/compare/best-open-source-llm-observability-tools ## On this page - [1. PostHog](#1-posthog) - [2. Langfuse](#2-langfuse) - [3. Opik](#3-opik) - [4. OpenLLMetry](#4-openllmetry) - [5. Phoenix](#5-phoenix) - [Other open source LLMO tools worth knowing](#other-open-source-llmo-tools-worth-knowing) - [Which LLM observability tool should you choose?](#which-llm-observability-tool-should-you-choose) - [Is PostHog right for you?](#is-posthog-right-for-you) - [Frequently asked questions](#frequently-asked-questions) - [Related reading](#related-reading) - [Ian Vanagas](https://posthog.com/community/profiles/29296.md) - [Natalia Amorim](https://posthog.com/community/profiles/35321.md) Nov 16, 2026 - [Roundups](https://posthog.com/compare/roundups.md) To build LLM-powered apps, developers need to know how users are using their app. LLM observability tools help them do this by capturing LLM provider requests and generations, then visualizing and aggregating them. This helps developers monitor, debug, and improve their apps. To help you pick the best of these tools, we put together this list. All of the following products: 1. Integrate with popular LLM providers and frameworks like OpenAI, Anthropic, and Vercel AI SDK to capture generations. 2. Let you view individual generations and traces from your app. 3. Calculate metrics such as cost, latency, and more, which can either be visualized in a dashboard or queried by your agents. 4. Are open source (or source-available) and self-hostable. 5. Have a free hosted version (minus one of them…) **What are the best open source LLM observability tools? The short version** | Tool | License | Free hosted plan | | --- | --- | --- | | **PostHog** | MIT | 100k events a month, unlimited seats, 1-year retention (30 days for prompts and outputs) | | **Langfuse** | MIT | 50k units a month, 2 users, 30-day data access | | **Opik** | Apache 2.0 | 25k spans a month, 10 team members, 60-day retention | | **OpenLLMetry** | Apache 2.0 | Via Traceloop: 50k spans a month, 5 seats, 24-hour retention | | **Phoenix** | Elastic License 2.0 | None for Phoenix itself; Arize AX Free has 25k spans a month | **PostHog** is the pick if you want LLM traces tied to the rest of your product data. **Langfuse** is the most complete dedicated platform. **Opik** suits teams that train models as well as build on them. **OpenLLMetry** is instrumentation rather than a full platform, and fits best if you already run OpenTelemetry. **Phoenix** is free to self-host with no feature gates, though its license is source-available. If open source is not a requirement, check out our [best AI observability tools](https://posthog.com/compare/best-ai-observability-tools.md) roundup for a broader list of options. ## 1\. PostHog - **Who it's for:** teams that want [LLM traces](https://posthog.com/docs/ai-observability/traces.md) tied to the same users as their [product analytics](https://posthog.com/docs/product-analytics.md), [session replays](https://posthog.com/docs/session-replay.md), [experiments](https://posthog.com/docs/experiments.md), [errors](https://posthog.com/docs/error-tracking.md), and more, and want their agents to be able to [query all of it](https://posthog.com/docs/model-context-protocol.md). - **Who it isn't for:** teams that need a built-in LLM gateway or proxy, or want to self-host at scale. - **License:** MIT (enterprise features are licensed separately) - [PostHog on GitHub](https://github.com/PostHog/posthog) ![PostHog](https://res.cloudinary.com/dmukukwp6/image/upload/w_1600,c_limit,q_auto,f_auto/Clean_Shot_2026_11_17_at_10_40_36_2x_ba54396106.png) [PostHog](https://posthog.com/.md) is your product's context layer. It combines [AI observability](https://posthog.com/ai-observability.md) with [product](https://posthog.com/product-analytics.md) and [web analytics](https://posthog.com/web-analytics.md), [session replay](https://posthog.com/session-replay.md), [feature flags](https://posthog.com/feature-flags.md), [experiments](https://posthog.com/experiments.md), [error tracking](https://posthog.com/error-tracking.md), [logs](https://posthog.com/logs.md), and [surveys](https://posthog.com/surveys.md), and a [data warehouse](https://posthog.com/warehouse.md) and [CDP](https://posthog.com/cdp.md) connect it all to the business data you keep elsewhere. Its LLM observability product, [AI observability](https://posthog.com/ai-observability.md), captures details of generations, provides an aggregated metrics dashboard, and more. It has [integrations](https://posthog.com/docs/ai-observability/installation.md) for more than 40 providers, frameworks, and gateways, including OpenAI, Anthropic, LangChain, and the Vercel AI SDK, plus OpenTelemetry, and it rolls everything up into an aggregated metrics dashboard. #### What does PostHog AI observability include? - **Tracing and costs:** Captures every [generation](https://posthog.com/docs/ai-observability/generations.md) and [trace](https://posthog.com/docs/ai-observability/traces.md) with its tokens, cost, latency, and errors, and rolls them up in a [dashboard](https://posthog.com/docs/ai-observability/dashboard.md). - **Product context:** LLM data sits alongside your product and business data, so you can query it with [SQL](https://posthog.com/docs/data-warehouse/sql.md), watch [session replays](https://posthog.com/docs/session-replay.md) of AI interactions, and [A/B test prompts](https://posthog.com/tutorials/llm-ab-tests.md). - **[Prompt management](https://posthog.com/docs/prompt-management.md):** Create and version prompts directly in PostHog. Prompts are fetched at runtime via the SDK with caching and fallback support, so you can update them without code deploys. Non-engineers can iterate on prompts from the UI, and every change creates an immutable version you can compare, restore, or link to traces to see which prompt versions drive which outputs. - **[Evaluations](https://posthog.com/docs/ai-evals.md):** Score production generations with LLM-as-a-judge, code-based checks, or sentiment analysis, and queue traces for [human review](https://posthog.com/docs/ai-observability/trace-reviews.md). This tracks quality over time – not just whether API calls succeed, but [whether they're actually good](https://posthog.com/blog/stop-ai-slop.md). - **Agent access:** The [MCP server](https://posthog.com/docs/ai-observability/surfaces/mcp.md) lets Claude Code, Cursor, or Codex check costs, errors, and traces from your editor, and you can ask the same questions from [Slack](https://posthog.com/slack.md) or the [CLI](https://posthog.com/docs/cli.md). #### How much does PostHog AI observability cost? PostHog Cloud is free to start: 100k AI observability events every month, with unlimited seats and 1-year retention (prompts and outputs are kept for 30 days). Beyond this, pricing is usage-based and [totally transparent](https://posthog.com/pricing.md). You can also [self-host PostHog](https://posthog.com/docs/self-host.md) for free under the MIT license (free-plan features only), though we recommend Cloud. Use the [setup wizard](https://posthog.com/wizard.md) to get started in minutes – no sales call or elaborate configuration needed. ![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_9608fcca70) ![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_dark_a92b0e022d) Add LLM observability in one command The wizard finds your LLM calls and wires them up for you. `npx @posthog/wizard ai-observability` ![PostHog Wizard hedgehog](https://res.cloudinary.com/dmukukwp6/image/upload/wizard_3f8bb7a240.png) ## 2\. Langfuse - **Who it's for:** teams that want a complete and dedicated LLM engineering platform, self-hosted or managed. - **Who it isn't for:** teams that want product analytics, session replay, or experiments in the same tool, or more than two users on the free cloud plan. - **License:** MIT (enterprise features are licensed separately) - [Langfuse on GitHub](https://github.com/langfuse/langfuse) [Langfuse](https://posthog.com/compare/posthog-vs-langfuse.md) (recently acquired by ClickHouse) is an open source LLM engineering platform. It provides LLM call tracking and tracing, prompt management, evaluation, datasets, and more. These give LLM app developers tools they need for their entire workflow. ClickHouse acquired Langfuse in January 2026. Both companies say the core stays MIT-licensed and Langfuse Cloud keeps running as a standalone service. We compare it with PostHog in detail in [PostHog vs Langfuse](https://posthog.com/compare/posthog-vs-langfuse.md). #### What does Langfuse include? - **Tracing:** LLM call tracking and tracing with session tracking, native SDKs for Python and JavaScript, and integrations with most LLM providers and agent frameworks. - **OpenTelemetry:** Can act as an OpenTelemetry backend, so any OTel-instrumented app can send it traces. - **Evaluation:** Scores, datasets, and annotation queues for human review, plus prompt management and a playground. - **Breadth:** Langfuse lists dozens of features, from session tracking to batch exports to SOC 2 reports. - **AI and agent tools:** An MCP server with read and write tools, an in-app assistant called Halo, and an AI gateway in public beta. #### How much does Langfuse cost? Langfuse can be self-hosted for free, though some admin and security features, like audit logs and project-level roles, need an enterprise license key. If you prefer a managed service, Langfuse Cloud is free to use up to 50k units per month and 2 users, but this only includes 30 days of data access. Pricing beyond this starts at $29/month for 100k units, unlimited users, and 90 days of data access. Additional usage starts at $8 per 100k units and decreases with volume, down to $6 per 100k units above 50M units. ## 3\. Opik - **Who it's for:** teams that train or fine-tune their own models as well as build LLM apps, and want a platform that's Apache 2.0 throughout. - **Who it isn't for:** teams with high trace volume that want to stay on a free cloud plan, which is capped at 25k spans a month. - **License:** Apache 2.0 - [Opik on GitHub](https://github.com/comet-ml/opik) **Opik** is an open source platform for evaluating, testing, and monitoring LLM apps. It provides tracing, annotations, a prompt and model playground, evaluation, and more. It's built by Comet, an end-to-end model evaluation platform for developers. In 2026, Opik 2.0 shifted its focus toward agents, adding test suites and an agent playground, and Comet later added cost tracking for coding agents like Claude Code and Codex. #### What does Opik include? - **Tracing:** Traces LLM calls and agent workflows, with annotations for reviewing them. - **Evaluation:** Automated evaluations, online evaluation rules, and a prompt and model playground. - **Monitoring:** Dashboards and alerts for production traffic. - **Agent tools:** An agent optimizer, guardrails, and its own MCP server. - **Comet platform:** Comet's experiment tracking and model registry make it ideal for teams training and hosting models of their own, not just using the LLM providers. #### How much does Opik cost? Opik's free hosted plan provides 25k spans per month with up to 10 team members and 60-day data retention. Beyond this, its Pro plan is $19 per month for 100k spans per month and up to 50 team members, with every extra 100k spans costing $5. Self-hosting is free, and the whole platform is Apache 2.0, with no separately licensed enterprise code. ## 4\. OpenLLMetry - **Who it's for:** teams already running OpenTelemetry that want LLM instrumentation feeding the observability stack they have. - **Who it isn't for:** teams that want a full platform with prompt management built in, or more than 24 hours of retention on Traceloop's free plan. - **License:** Apache 2.0 - [OpenLLMetry on GitHub](https://github.com/traceloop/openllmetry) **OpenLLMetry** is an open-source set of LLM instrumentations built on OpenTelemetry. It doesn't have its own UI, so you view the data in Traceloop or another backend. It was built by Traceloop and recommends using its SDK to capture data. ServiceNow announced it was acquiring Traceloop in March 2026, and has since made its platform part of ServiceNow’s AI Control Tower. OpenLLMetry itself remains open source and is still shipping releases to the Python package. #### What does OpenLLMetry include? - **Instrumentation:** Captures data from more than a dozen LLM providers, plus vector DBs and LLM frameworks, through OpenTelemetry. - **Destinations:** Sends that data to more than 20 supported destinations, from Traceloop to [Datadog](https://posthog.com/compare/best-datadog-alternatives.md) to Honeycomb, so it's very likely to work with the observability tools you already use. - **Full-stack traces:** As part of the broader OpenTelemetry ecosystem, it can instrument things like your database and API calls too. - **Standards:** Its semantic conventions for LLMs were the starting point for OpenTelemetry's own GenAI conventions. - **MCP:** Instruments MCP calls so they show up in your traces. #### How much does OpenLLMetry cost? OpenLLMetry itself is free. Traceloop, the hosted platform, is free up to 50k spans per month and 5 seats, but this only provides 24-hour data retention. For more, you'll need to talk to sales, which now means ServiceNow. ## 5\. Phoenix - **Who it's for:** teams that want to self-host with no feature gates on an OpenTelemetry-based setup, or expect to move to Arize AX later. - **Who it isn't for:** teams that need an OSI-approved license, or a free hosted version of Phoenix itself. - **License:** Elastic License 2.0 (source-available) - [Phoenix on GitHub](https://github.com/Arize-ai/phoenix) Phoenix is a source-available AI observability platform. It provides tracing, evaluation, datasets and experiments, a playground, prompt management, and more. It works out-of-the-box with frameworks like LlamaIndex and LangChain as well as LLM providers like OpenAI, Bedrock, and more. It's built by Arize AI, a unified AI observability and evaluation platform that Dynatrace acquired in October 2026. One caveat on "open source": Phoenix uses the Elastic License 2.0. You can self-host it for free with no feature gates, but it isn't an OSI-approved license, and you can't offer Phoenix to others as a managed service. #### What does Phoenix include? - **Tracing:** Built on OpenTelemetry and OpenInference, a set of conventions and plugins that are complementary to OpenTelemetry, so it fits into your existing telemetry stack. - **Evaluation and experiments:** Evaluations, datasets and experiments, a playground, and prompt management. - **Agent tools:** A built-in AI engineering agent called PXI and a remote MCP server, both added in 2026. - **Arize AX:** Connected to a broader platform that also monitors traditional ML and computer vision models, which helps teams running more than LLMs. #### How much does Phoenix cost? Phoenix is free to self-host. Arize no longer advertises a free hosted Phoenix, although existing Phoenix Cloud users can still log in. For a managed option, it points you to Arize AX, its separate SaaS platform: AX Free includes 25k spans per month with 15-day retention, and AX Pro costs $50 per month for 50k spans with 30-day retention. Both include unlimited users. Dynatrace says Phoenix and AX will both continue, with Arize’s capabilities integrated into Dynatrace over time. ## Other open source LLMO tools worth knowing These didn't make the list, either because they miss one of the criteria above or because they're newer and smaller: - **MLflow** – One of the largest open source ML projects, with an Apache 2.0 license and a home at the Linux Foundation. Its tracing supports 40+ frameworks and providers. There's no free hosted version from the project itself; managed MLflow runs on Databricks and AWS. - **LangWatch** – Open core (Apache 2.0, with separately licensed enterprise code) and known for agent simulation testing. Its free cloud plan covers 50k events a month for 2 users, with 14-day retention. - **Laminar** – Open core (Apache 2.0, with an Elastic License 2.0 enterprise edition), with tracing and dashboards for cost, tokens, and latency. The free cloud plan is small: 1 GB of data, 7-day retention, and one seat. - **Lunary** – An observability and prompt management platform built for LLM chatbots; its repository is no longer public, self-hosting now needs an enterprise license, and there's no free plan, so we removed it. LangSmith and Braintrust come up a lot too, but neither is open source. We cover them in [the best AI observability tools](https://posthog.com/compare/best-ai-observability-tools.md). **A note on Helicone** [Helicone](https://posthog.com/compare/best-helicone-alternatives.md) was on earlier versions of this list. It's still open source (Apache 2.0) and you can self-host it, but Mintlify acquired it in March 2026 and put it into maintenance mode, and new sign-ups to the hosted product are disabled. In November 2026, its site announced that it's now part of Experiential Labs, with details to come. If you're on Helicone today, see our [Helicone alternatives](https://posthog.com/compare/best-helicone-alternatives.md). ## Which LLM observability tool should you choose? - Want LLM traces connected to the rest of your product data, from events to replays to errors, and queryable by your agents through the MCP? **PostHog** - Need a dedicated, fully-featured LLM observability platform you can self-host? **Langfuse** - Building or fine-tuning models as well as LLM apps? **Opik** - Already using OpenTelemetry and want LLM instrumentation to fit into your existing stack? **OpenLLMetry** - Need AI observability beyond LLMs, including ML models and computer vision? **Phoenix** (with Arize AX) ## Is PostHog right for you? Here's the (short) sales pitch. We're biased, obviously, but we think you'll love PostHog if: - You want LLM observability connected to the rest of your product data – session replays, feature flags, A/B testing, and analytics all in one place - You're already using PostHog, so adding AI Observability requires no new vendor or contract - You want your coding agent to check LLM costs, errors, and traces through [the MCP](https://posthog.com/mcp.md) - You want to try before you buy (we're self-serve with a [generous free tier](https://posthog.com/pricing.md)) It's completely free to get started – no credit card required. Our [setup wizard](https://posthog.com/wizard.md) handles configuration in minutes, or you can check out [our docs](https://posthog.com/docs/ai-observability.md) to do it yourself. ![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_9608fcca70) ![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_dark_a92b0e022d) Add LLM observability in one command The wizard finds your LLM calls and wires them up for you. `npx @posthog/wizard ai-observability` ![PostHog Wizard hedgehog](https://res.cloudinary.com/dmukukwp6/image/upload/wizard_3f8bb7a240.png) ## Frequently asked questions **What does it mean for PostHog to be your product's context layer?** It means your LLM traces sit next to everything else you know about the user. [AI observability](https://posthog.com/ai-observability.md), [product analytics](https://posthog.com/product-analytics.md), [session replay](https://posthog.com/session-replay.md), [error tracking](https://posthog.com/error-tracking.md), [feature flags](https://posthog.com/feature-flags.md), and [logs](https://posthog.com/logs.md) all share one model of users and sessions. For any generation, you can see who the user was, what they did next, whether an error fired, and which experiment they were in, etc. The [data warehouse](https://posthog.com/warehouse.md) and [data pipelines](https://posthog.com/data-pipelines.md) extend it to the business data that lives elsewhere, like Stripe, HubSpot, and Postgres. So an agent working from [Slack](https://posthog.com/slack.md), the [CLI](https://posthog.com/docs/cli.md), or your editor via [the MCP](https://posthog.com/mcp.md) can answer questions about your product data without stitching a bunch of tools together. **PostHog says it makes your product "self-driving" – what does that mean?** It means PostHog digs through your product data, finds what's worth fixing, and has agents do the work. It starts with context. A full suite of developer tools – [AI observability](https://posthog.com/ai-observability.md), [product analytics](https://posthog.com/product-analytics.md), [session replay](https://posthog.com/session-replay.md), [feature flags](https://posthog.com/feature-flags.md), [experiments](https://posthog.com/experiments.md), [error tracking](https://posthog.com/error-tracking.md), [logs](https://posthog.com/logs.md), and more – captures everything happening in your product, and a [data warehouse](https://posthog.com/warehouse.md) unifies it into one source agents can read across. From there, [Scouts](https://posthog.com/blog/what-is-a-scout.md) read across all of it and sort what's worth knowing from what's just noise. What clears the bar becomes a report in your inbox: an agent picks it up, roots out the cause, and opens a PR when the fix is clear. You review and merge. You can steer it from [Slack](https://posthog.com/slack.md), the [web app](https://posthog.com/ai.md), or your own editor via [the MCP](https://posthog.com/mcp.md) or [CLI](https://posthog.com/docs/cli.md). **What is LLM observability?** LLM observability (also called AI observability) is the practice of monitoring and understanding how your LLM-powered application behaves in production. It typically includes capturing individual LLM calls (inputs, outputs, latency, token usage), aggregating metrics across requests, and providing tools to debug issues and improve model performance. It's similar to traditional application observability, but focused on the unique characteristics of LLM systems – non-deterministic outputs, high token costs, prompt sensitivity, and the challenge of evaluating quality. **What features do you need in an LLM observability tool?** A good LLM observability tool gives you visibility into how your AI-powered app is performing in production. Most solid tools include: - **Tracing and logging** to capture individual LLM calls, inputs, outputs, and latency - **Cost tracking** to monitor token usage and spend across providers and models - **Aggregated dashboards** for monitoring performance trends over time - **Self-hosting options** so you keep full control of your data and model inputs More advanced tools go further with: - **Prompt management** for versioning, testing, and deploying prompts - **Evaluation and evals** to score model outputs automatically or with human review - **Datasets** for curating examples and running regression tests - **Integration with product analytics** so you can connect LLM performance to user behavior - **OpenTelemetry compatibility** for teams with existing observability infrastructure **When should you consider an LLM observability tool?** If you're building an LLM-powered app and have shipped to real users, you need one. Common signals that you're ready: - You're not sure which prompts or models are causing user drop-off - You're spending more on tokens than expected and don't know where the cost is going - You have no visibility into latency spikes or failure rates - You want to [run evals or compare model versions systematically](https://posthog.com/blog/testing-ai-agents.md) Most tools on this list are free to start, so there's no reason to wait. **Which open source LLM observability tools have an MCP server?** PostHog, Langfuse, Opik, and Phoenix all have one, so a coding agent like Claude Code or Cursor can query your traces and costs directly: - **PostHog** – The [MCP server](https://posthog.com/docs/ai-observability/surfaces/mcp.md) covers AI observability along with the rest of the platform, so an agent can go from a trace to the session replay or error behind it. - **Langfuse** – A native MCP server with read and write tools for prompts, scores, and datasets, plus read access to observations. - **Opik** and **Phoenix** – Each ships its own MCP server. OpenLLMetry takes a different angle: it instruments MCP calls so they show up in your traces. Traceloop also has a separate MCP server for querying OpenTelemetry traces. **Do I need a separate LLM observability tool if I already use PostHog?** No. PostHog's [LLM observability](https://posthog.com/docs/ai-observability.md) product is built into the platform, so if you're already using PostHog for product analytics or session replay, you can add LLM observability without a new vendor or contract. You get 100k LLM events free per month. [Getting started](https://posthog.com/docs/ai-observability/start-here.md) is easy; once you install the SDK, it will handle all the heavy lifting. Swap in our wrapped LLM client and we'll capture your generations automatically. **What's the difference between LLM observability and traditional application monitoring?** **Traditional application monitoring** focuses on things like error rates, latency, and uptime – binary metrics where something either works or doesn't. **LLM observability** adds a quality dimension: you need to evaluate whether model outputs are actually good, not just whether the API call succeeded. This is why tools like Langfuse and Opik invest heavily in evals, human review, and prompt management – capabilities that don't exist in [traditional APM tools](https://posthog.com/compare/best-apm-tools-for-developers.md). **Are these tools compatible with all LLM providers?** Most tools on this list support the major providers – OpenAI, Anthropic, Google Gemini, and AWS Bedrock – as well as popular frameworks like LangChain, LlamaIndex, and Vercel AI SDK. Coverage varies by tool. For specific provider support, check each tool's documentation. ## Related reading - [PostHog AI observability documentation](https://posthog.com/docs/ai-observability.md) - [How to A/B test LLM models and prompts](https://posthog.com/tutorials/llm-ab-tests.md) - [The best AI observability tools](https://posthog.com/compare/best-ai-observability-tools.md) for more options - [The cheapest AI observability tools](https://posthog.com/compare/cheapest-ai-observability-tools.md) for budget-conscious teams - [PostHog vs Langfuse](https://posthog.com/compare/posthog-vs-langfuse.md) and the [best Langfuse alternatives](https://posthog.com/compare/best-langfuse-alternatives.md) - [The best error tracking tools](https://posthog.com/compare/best-error-tracking-tools.md) if you're also instrumenting the rest of your app alongside the LLM parts ![](https://res.cloudinary.com/dmukukwp6/image/upload/icon_ae03b6df11) Subscribe to our newsletter #### build mode Read by 75,000+ founders and builders Subscribe We'll share your email with Substack > PostHog is your product's context layer. With a full suite of developer tools – [AI observability](https://posthog.com/ai-observability.md), [product analytics](https://posthog.com/product-analytics.md), [session replay](https://posthog.com/session-replay.md), [feature flags](https://posthog.com/feature-flags.md), [experiments](https://posthog.com/experiments.md), [error tracking](https://posthog.com/error-tracking.md), [logs](https://posthog.com/logs.md), and more – PostHog ingests and stores all the data agents need to diagnose problems, uncover opportunities, and ship fixes. A [data warehouse](https://posthog.com/warehouse.md) and [CDP](https://posthog.com/data-pipelines.md) tie it all together, unifying that context into one source agents can read across. You can steer it all from [Slack](https://posthog.com/slack.md), [the web app](https://posthog.com/ai.md), the desktop ([PostHog Desktop](https://posthog.com/desktop.md)), or your own editor via [the MCP](https://posthog.com/mcp.md).