Linking Confluent Cloud as a source
This source is currently in alpha. The interface and available tables may change.
Let AI connect your sources for you
Skip the manual setup — run this in your project and the wizard auto-detects your databases and APIs and connects them to PostHog.
Contents
The Confluent Cloud connector syncs operational metrics from the Confluent Cloud Metrics API into the PostHog Data warehouse: Kafka throughput, consumer lag, connector activity, ksqlDB, Schema Registry, and Flink compute pool utilization. Confluent only retains this data for about 7 days, so warehousing it lets you do capacity planning and cost analysis over longer time ranges.
Prerequisites
You need a Confluent Cloud service account with the MetricsViewer role and a Cloud API key owned by it. Cluster-scoped API keys don't work with the Metrics API, so make sure you create a Cloud (resource management) key.
- In the Confluent Cloud console, go to Administration → Accounts & access and create a service account.
- Grant it the MetricsViewer role for the resources you want to collect metrics for.
- Go to Administration → API keys and create a Cloud API key owned by that service account. Note the key and secret.
Adding a data source
- In PostHog, go to the Sources tab of the data pipeline section.
- Click + New source and click Link next to this source.
- Enter your credentials (see Configuration below) and click Next.
- Select the tables you want to sync, choose a sync method and frequency, then click Import.
Once the syncs are complete, you can start querying this data in PostHog.
When linking Confluent Cloud, you'll need:
- Cloud API key and Cloud API secret – from the steps above.
- Resource IDs – the IDs of the resources to collect metrics for, as comma-separated lists. For example, Kafka cluster IDs (
lkc-...) are shown under Cluster settings, connector IDs (lcc-...) under the connector's settings. Leave a resource type blank to skip it.
Sync modes
The metrics tables sync incrementally on the timestamp column: each sync only queries the time window since the last one, plus a small overlap to pick up late-arriving data. Because the Metrics API retains roughly 7 days of data, keep a frequent sync schedule so no window is missed.
Each table can be synced in one of several modes, depending on what the source supports:
- Webhook (when available) – the source pushes changes to PostHog in real time. Fastest freshness, lowest ongoing cost, and the only mode that reliably captures updates and deletes.
- Incremental – only new or updated rows are synced on each run, using a cursor field (such as an
updated_attimestamp). Cheaper than a full refresh, but deletes aren't captured. - Append only – new rows are appended using a cursor field; existing rows are never updated. Ideal for immutable, append-only tables like event logs.
- Full refresh – the whole table is reloaded on every sync. Use it when a table has no reliable cursor or when you need deletions reflected.
See sync methods for a full explanation of how each mode works and how to choose between them.
Configuration
| Option | Type | Required |
|---|---|---|
Cloud API key | text | Yes |
Cloud API secret | password | Yes |
Kafka cluster IDs | text | No |
Connector IDs | text | No |
ksqlDB cluster IDs | text | No |
Schema Registry cluster IDs | text | No |
Flink compute pool IDs | text | No |
Supported tables
The metrics tables contain one row per metric, resource, and UTC hour, with the metric's aggregated value. Values are aggregated per resource, so per-topic or per-consumer-group detail isn't included. The metric_descriptors table documents every metric name, its unit, and its data type.
| Table | Description | Sync method | Incremental field | Primary key |
|---|---|---|---|---|
metric_descriptors | Metadata catalog for every metric available in the Metrics API 'cloud' dataset, including its data type, unit, and lifecycle stage. | Full refresh | — | — |
resource_descriptors | Metadata catalog for the resource types that metrics can be scoped to, such as Kafka clusters, connectors, and Flink compute pools. | Full refresh | — | — |
kafka_metrics | Hourly time-series values for every kafka metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
connector_metrics | Hourly time-series values for every connector metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
ksqldb_metrics | Hourly time-series values for every ksql metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
schema_registry_metrics | Hourly time-series values for every schema_registry metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
compute_pool_metrics | Hourly time-series values for every compute_pool metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
Troubleshooting
- If you see an authentication error, the API key or secret is invalid or has been revoked. Create a new Cloud API key and reconnect.
- If you see a permissions error, the key authenticated but isn't authorized for the configured resources. Make sure it's a Cloud API key (not a cluster-scoped key) owned by a service account with the MetricsViewer role for those resources.
- If a metrics table syncs no rows, check that the matching resource IDs are filled in on the source settings and that the resources produced data in the last 7 days.
If your sync is failing or data looks wrong, see the Data warehouse troubleshooting guide. If that doesn't help, contact support – we're happy to help.