Linking Confluent Cloud as a source
This source is currently in alpha. The interface and available tables may change.
The Confluent Cloud connector syncs operational metrics from the Confluent Cloud Metrics API into the PostHog Data warehouse: Kafka throughput, consumer lag, connector activity, ksqlDB, Schema Registry, and Flink compute pool utilization. Confluent only retains this data for about 7 days, so warehousing it lets you do capacity planning and cost analysis over longer time ranges.
Prerequisites
You need a Confluent Cloud service account with the MetricsViewer role and a Cloud API key owned by it. Cluster-scoped API keys don't work with the Metrics API, so make sure you create a Cloud (resource management) key.
- In the Confluent Cloud console, go to Administration → Accounts & access and create a service account.
- Grant it the MetricsViewer role for the resources you want to collect metrics for.
- Go to Administration → API keys and create a Cloud API key owned by that service account. Note the key and secret.
Adding a data source
- In PostHog, go to the Sources tab of the data pipeline section.
- Click + New source and click Link next to this source.
- Enter your credentials (see Configuration below) and click Next.
- Select the tables you want to sync, choose a sync method and frequency, then click Import.
Once the syncs are complete, you can start querying this data in PostHog.
When linking Confluent Cloud, you'll need:
- Cloud API key and Cloud API secret – from the steps above.
- Resource IDs – the IDs of the resources to collect metrics for, as comma-separated lists. For example, Kafka cluster IDs (
lkc-...) are shown under Cluster settings, connector IDs (lcc-...) under the connector's settings. Leave a resource type blank to skip it.
Sync modes
The metrics tables sync incrementally on the timestamp column: each sync only queries the time window since the last one, plus a small overlap to pick up late-arriving data. Because the Metrics API retains roughly 7 days of data, keep a frequent sync schedule so no window is missed.
Each table can be synced in one of several modes, depending on what the source supports:
- Webhook (when available) – the source pushes changes to PostHog in real time. Fastest freshness, lowest ongoing cost, and the only mode that reliably captures updates and deletes.
- Incremental – only new or updated rows are synced on each run, using a cursor field (such as an
updated_attimestamp). Cheaper than a full refresh, but deletes aren't captured. - Append only – new rows are appended using a cursor field; existing rows are never updated. Ideal for immutable, append-only tables like event logs.
- Full refresh – the whole table is reloaded on every sync. Use it when a table has no reliable cursor or when you need deletions reflected.
See sync methods for a full explanation of how each mode works and how to choose between them.
Configuration
| Option | Type | Required |
|---|---|---|
Cloud API key | text | Yes |
Cloud API secret | password | Yes |
Kafka cluster IDs | text | No |
Connector IDs | text | No |
ksqlDB cluster IDs | text | No |
Schema Registry cluster IDs | text | No |
Flink compute pool IDs | text | No |
Supported tables
The metrics tables contain one row per metric, resource, and UTC hour, with the metric's aggregated value. Values are aggregated per resource, so per-topic or per-consumer-group detail isn't included. The metric_descriptors table documents every metric name, its unit, and its data type.
| Table | Description | Sync method | Incremental field | Primary key |
|---|---|---|---|---|
metric_descriptors | Metadata catalog for every metric available in the Metrics API 'cloud' dataset, including its data type, unit, and lifecycle stage. | Full refresh | — | — |
resource_descriptors | Metadata catalog for the resource types that metrics can be scoped to, such as Kafka clusters, connectors, and Flink compute pools. | Full refresh | — | — |
kafka_metrics | Hourly time-series values for every kafka metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
connector_metrics | Hourly time-series values for every connector metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
ksqldb_metrics | Hourly time-series values for every ksql metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
schema_registry_metrics | Hourly time-series values for every schema_registry metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
compute_pool_metrics | Hourly time-series values for every compute_pool metric in the Metrics API catalog, aggregated per resource. The API retains about 7 days of data, so the first sync backfills that window and regular syncs keep history beyond it. | Incremental, Full refresh | timestamp | — |
Troubleshooting
- If you see an authentication error, the API key or secret is invalid or has been revoked. Create a new Cloud API key and reconnect.
- If you see a permissions error, the key authenticated but isn't authorized for the configured resources. Make sure it's a Cloud API key (not a cluster-scoped key) owned by a service account with the MetricsViewer role for those resources.
- If a metrics table syncs no rows, check that the matching resource IDs are filled in on the source settings and that the resources produced data in the last 7 days.
If your sync is failing or data looks wrong, see the Data warehouse troubleshooting guide. If that doesn't help, contact support – we're happy to help.