> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

# Linking Confluent Cloud as a source

Canonical URL: https://posthog.com/docs/data-pipelines/sources/confluent-cloud

> **Cloud region:** This page uses US Cloud hosts. For an EU Cloud project, use `https://eu.posthog.com` instead of `https://us.posthog.com` or `https://app.posthog.com` (app host for links).

## On this page

- [Prerequisites](#prerequisites)
- [Adding a data source](#adding-a-data-source)
- [Sync modes](#sync-modes)
- [Configuration](#configuration)
- [Supported tables](#supported-tables)
- [Troubleshooting](#troubleshooting)

**Beta release**

This source is currently in **beta**. It works end to end, but you may still hit rough edges. [Let us know](https://us.posthog.com/#panel=support%3Asupport%3Adata_warehouse%3A%3Atrue) if you do.

![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_9608fcca70)

![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_dark_a92b0e022d)

Let AI connect your sources for you

Skip the manual setup — run this in your project and the wizard auto-detects your databases and APIs and connects them to PostHog.

`npx @posthog/wizard warehouse`

![PostHog Wizard hedgehog](https://res.cloudinary.com/dmukukwp6/image/upload/wizard_3f8bb7a240.png)

![](https://res.cloudinary.com/dmukukwp6/image/upload/wizard_3f8bb7a240.png)Let AI connect your sources for you

The Confluent Cloud connector syncs operational metrics from the [Confluent Cloud Metrics API](https://api.telemetry.confluent.cloud/docs) into the PostHog Data warehouse: Kafka throughput, consumer lag, connector activity, ksqlDB, Schema Registry, and Flink compute pool utilization. Confluent only retains this data for about 7 days, so warehousing it lets you do capacity planning and cost analysis over longer time ranges.

## Prerequisites

You need a Confluent Cloud **service account** with the **MetricsViewer** role and a **Cloud API key** owned by it. Cluster-scoped API keys don't work with the Metrics API, so make sure you create a Cloud (resource management) key.

1.  In the [Confluent Cloud console](https://confluent.cloud), go to **Administration → Accounts & access** and create a service account.
2.  Grant it the **MetricsViewer** role for the resources you want to collect metrics for.
3.  Go to **Administration → API keys** and create a **Cloud API key** owned by that service account. Note the key and secret.

## Adding a data source

1.  In PostHog, go to the [Sources tab](https://app.posthog.com/data-management/sources) of the data pipeline section.
2.  Click **\+ New source** and click **Link** next to this source.
3.  Enter your credentials (see [Configuration](#configuration) below) and click **Next**.
4.  Select the tables you want to sync, choose a sync method and frequency, then click **Import**.

Once the syncs are complete, you can start querying this data in PostHog.

When linking Confluent Cloud, you'll need:

-   **Cloud API key** and **Cloud API secret** – from the steps above.
-   **Resource IDs** – the IDs of the resources to collect metrics for, as comma-separated lists. For example, Kafka cluster IDs (`lkc-...`) are shown under **Cluster settings**, connector IDs (`lcc-...`) under the connector's settings. Leave a resource type blank to skip it.

## Sync modes

The metrics tables sync incrementally on the `timestamp` column: each sync only queries the time window since the last one, plus a small overlap to pick up late-arriving data. Because the Metrics API retains roughly 7 days of data, keep a frequent sync schedule so no window is missed.

Each table can be synced in one of several modes, depending on what the source supports:

-   **Webhook** (when available) – the source pushes changes to PostHog in real time. Fastest freshness, lowest ongoing cost, and the only mode that reliably captures updates and deletes.
-   **Incremental** – only new or updated rows are synced on each run, using a cursor field (such as an `updated_at` timestamp). Cheaper than a full refresh, but deletes aren't captured.
-   **Append only** – new rows are appended using a cursor field; existing rows are never updated. Ideal for immutable, append-only tables like event logs.
-   **Full refresh** – the whole table is reloaded on every sync. Use it when a table has no reliable cursor or when you need deletions reflected.

See [sync methods](https://posthog.com/docs/data-pipelines/sources.md#sync-methods) for a full explanation of how each mode works and how to choose between them.

## Configuration

| Option | Type | Required |
| --- | --- | --- |
| `Cloud API key` | text | Yes |
| `Cloud API secret` | password | Yes |
| `Kafka cluster IDs` | text | No |
| `Connector IDs` | text | No |
| `ksqlDB cluster IDs` | text | No |
| `Schema Registry cluster IDs` | text | No |
| `Flink compute pool IDs` | text | No |

## Supported tables

The metrics tables contain one row per metric, resource, and UTC hour, with the metric's aggregated value. Values are aggregated per resource, so per-topic or per-consumer-group detail isn't included. The `metric_descriptors` table documents every metric name, its unit, and its data type.

| Table | Description | Sync method | Incremental field | Primary key |
| --- | --- | --- | --- | --- |
| `metric_descriptors` | Metadata catalog for every metric available in the Metrics API 'cloud' dataset, including its data type, unit, and lifecycle stage. | Full refresh | — | — |
| `resource_descriptors` | Metadata catalog for the resource types that metrics can be scoped to, such as Kafka clusters, connectors, and Flink compute pools. | Full refresh | — | — |
| `kafka_metrics` | Hourly time-series values for every kafka metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
| `connector_metrics` | Hourly time-series values for every connector metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
| `ksqldb_metrics` | Hourly time-series values for every ksql metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
| `schema_registry_metrics` | Hourly time-series values for every schema\_registry metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
| `compute_pool_metrics` | Hourly time-series values for every compute\_pool metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |

## Troubleshooting

-   If you see an authentication error, the API key or secret is invalid or has been revoked. Create a new Cloud API key and reconnect.
-   If you see a permissions error, the key authenticated but isn't authorized for the configured resources. Make sure it's a Cloud API key (not a cluster-scoped key) owned by a service account with the MetricsViewer role for those resources.
-   If a metrics table syncs no rows, check that the matching resource IDs are filled in on the source settings and that the resources produced data in the last 7 days.

If your sync is failing or data looks wrong, see the [Data warehouse troubleshooting guide](https://posthog.com/docs/data-warehouse/troubleshooting.md). If that doesn't help, [contact support](https://us.posthog.com/#panel=support%3Asupport%3Adata_warehouse%3A%3Atrue) – we're happy to help.