• Product
  • Pricing
  • Docs
  • Using PostHog
  • Community
  • Company
  • Login
  • Docs

  • Overview
    • Quickstart with PostHog Cloud
    • Overview
      • AWS
      • Azure
      • DigitalOcean
      • Google Cloud Platform
      • Hobby
      • Other platforms
      • Instance settings
      • Environment variables
      • Securing PostHog
      • Monitoring with Grafana
      • Running behind a proxy
      • Configuring email
      • Helm chart configuration
      • Deploying ClickHouse using Altinity.Cloud
      • Configuring Slack
      • Overview
        • Overview
        • Upgrade notes
        • Overview
        • 0001-events-sample-by
        • 0002_events_sample_by
        • 0003_fill_person_distinct_id2
        • ClickHouse
          • Backup
          • Kafka Engine
          • Resize disk
          • Restore
          • Vertical scaling
          • Horizontal scaling (Sharding & replication)
        • Kafka
          • Resize disk
          • Log retention
        • PostgreSQL
          • Resize disk
          • Troubleshooting long-running migrations
        • Plugin server
        • MinIO
        • Redis
        • Zookeeper
      • Disaster recovery
    • Troubleshooting and FAQs
    • Architecture
    • Managing hosting costs
    • EU-only hosting
    • Overview
    • Ingest live data
    • Ingest historical data
    • Identify users
    • User properties
    • Deploying a reverse proxy
    • Libraries
    • Badge
      • Snippet installation
      • Android
      • iOS
      • JavaScript
      • Flutter
      • React Native
      • Browser Extensions
      • Elixir
      • Go
      • Java
      • Node.js
      • PHP
      • Python
      • Ruby
      • Docusaurus v2
      • Gatsby
      • Google Tag Manager
      • Next.js
      • Nuxt.js
      • Retool
      • RudderStack
      • Segment
      • Sentry
      • Slack
      • Shopify
      • WordPress
      • Message formatting
      • Microsoft Teams
      • Slack
      • Discord
    • To another self-hosted instance
    • To PostHog from Amplitude
    • Between Cloud and self-hosted
    • Overview
    • Troubleshooting
      • Overview
      • Tutorial
      • TypeScript types
      • Developer reference
        • Amazon Kinesis Import
        • BitBucket Release Tracker
        • Braze Import
        • Event Replicator
        • GitHub Release Tracker
        • GitHub Star Sync
        • GitLab Release Tracker
        • Heartbeat
        • Ingestion Alert
        • Email Scoring
        • n8n Connector
        • Orbit Connector
        • Redshift Import
        • Segment Connector
        • Shopify Connector
        • Twitter Followers Tracker
        • Zendesk Connector
        • Airbyte Exporter
        • Amazon S3 Export
        • BigQuery Export
        • Customer.io Connector
        • Databricks Export
        • Engage Connector
        • GCP Pub/Sub Connector
        • Google Cloud Storage Export
        • Hubspot Connector
        • Intercom Connector
        • Migrator 3000
        • PagerDuty Connector
        • PostgreSQL Export
        • Redshift Export
        • RudderStack Export
        • Salesforce Connector
        • Sendgrid Connector
        • Sentry Connector
        • Snowflake Export
        • Twilio Connector
        • Variance Connector
        • Zapier Connector
        • Downsampler
        • Event Sequence Timer
        • First Time Event Tracker
        • Property Filter
        • Property Flattener
        • Schema Enforcer
        • Taxonomy Standardizer
        • Unduplicator
        • Automatic Cohort Creator
        • Currency Normalizer
        • GeoIP Enricher
        • Timestamp Parser
        • URL Normalizer
        • User Agent Populator
    • Overview
    • POST-only public endpoints
    • Actions
    • Annotations
    • Cohorts
    • Dashboards
    • Event definitions
    • Events
    • Experiments
    • Feature flags
    • Funnels
    • Groups
    • Groups types
    • Insights
    • Invites
    • Members
    • Persons
    • Plugin configs
    • Plugins
    • Projects
    • Property definitions
    • Session recordings
    • Trends
    • Users
    • Data model
    • Overview
    • GDPR guidance
    • HIPAA guidance
    • CCPA guidance
    • Data egress & compliance
    • Overview
    • Code of conduct
    • Recognizing contributions
  • Using PostHog

  • Table of contents
      • Dashboards
      • Funnels
      • Group Analytics
      • Insights
      • Lifecycle
      • Path analysis
      • Retention
      • Stickiness
      • Trends
      • Heatmaps
      • Session Recording
      • Correlation Analysis
      • Experimentation
      • Feature Flags
      • Actions
      • Annotations
      • Cohorts
      • Data Management
      • Events
      • Persons
      • Sessions
      • UTM segmentation
      • Team collaboration
      • Organizations & projects
      • Settings
      • SSO & SAML
      • Toolbar
      • Notifications & alerts
  • Tutorials
    • All tutorials
    • Actions
    • Apps
    • Cohorts
    • Dashboards
    • Feature flags
    • Funnels
    • Heatmaps
    • Path analysis
    • Retention
    • Session recording
    • Trends
  • Support
  • Docs

  • Overview
    • Quickstart with PostHog Cloud
    • Overview
      • AWS
      • Azure
      • DigitalOcean
      • Google Cloud Platform
      • Hobby
      • Other platforms
      • Instance settings
      • Environment variables
      • Securing PostHog
      • Monitoring with Grafana
      • Running behind a proxy
      • Configuring email
      • Helm chart configuration
      • Deploying ClickHouse using Altinity.Cloud
      • Configuring Slack
      • Overview
        • Overview
        • Upgrade notes
        • Overview
        • 0001-events-sample-by
        • 0002_events_sample_by
        • 0003_fill_person_distinct_id2
        • ClickHouse
          • Backup
          • Kafka Engine
          • Resize disk
          • Restore
          • Vertical scaling
          • Horizontal scaling (Sharding & replication)
        • Kafka
          • Resize disk
          • Log retention
        • PostgreSQL
          • Resize disk
          • Troubleshooting long-running migrations
        • Plugin server
        • MinIO
        • Redis
        • Zookeeper
      • Disaster recovery
    • Troubleshooting and FAQs
    • Architecture
    • Managing hosting costs
    • EU-only hosting
    • Overview
    • Ingest live data
    • Ingest historical data
    • Identify users
    • User properties
    • Deploying a reverse proxy
    • Libraries
    • Badge
      • Snippet installation
      • Android
      • iOS
      • JavaScript
      • Flutter
      • React Native
      • Browser Extensions
      • Elixir
      • Go
      • Java
      • Node.js
      • PHP
      • Python
      • Ruby
      • Docusaurus v2
      • Gatsby
      • Google Tag Manager
      • Next.js
      • Nuxt.js
      • Retool
      • RudderStack
      • Segment
      • Sentry
      • Slack
      • Shopify
      • WordPress
      • Message formatting
      • Microsoft Teams
      • Slack
      • Discord
    • To another self-hosted instance
    • To PostHog from Amplitude
    • Between Cloud and self-hosted
    • Overview
    • Troubleshooting
      • Overview
      • Tutorial
      • TypeScript types
      • Developer reference
        • Amazon Kinesis Import
        • BitBucket Release Tracker
        • Braze Import
        • Event Replicator
        • GitHub Release Tracker
        • GitHub Star Sync
        • GitLab Release Tracker
        • Heartbeat
        • Ingestion Alert
        • Email Scoring
        • n8n Connector
        • Orbit Connector
        • Redshift Import
        • Segment Connector
        • Shopify Connector
        • Twitter Followers Tracker
        • Zendesk Connector
        • Airbyte Exporter
        • Amazon S3 Export
        • BigQuery Export
        • Customer.io Connector
        • Databricks Export
        • Engage Connector
        • GCP Pub/Sub Connector
        • Google Cloud Storage Export
        • Hubspot Connector
        • Intercom Connector
        • Migrator 3000
        • PagerDuty Connector
        • PostgreSQL Export
        • Redshift Export
        • RudderStack Export
        • Salesforce Connector
        • Sendgrid Connector
        • Sentry Connector
        • Snowflake Export
        • Twilio Connector
        • Variance Connector
        • Zapier Connector
        • Downsampler
        • Event Sequence Timer
        • First Time Event Tracker
        • Property Filter
        • Property Flattener
        • Schema Enforcer
        • Taxonomy Standardizer
        • Unduplicator
        • Automatic Cohort Creator
        • Currency Normalizer
        • GeoIP Enricher
        • Timestamp Parser
        • URL Normalizer
        • User Agent Populator
    • Overview
    • POST-only public endpoints
    • Actions
    • Annotations
    • Cohorts
    • Dashboards
    • Event definitions
    • Events
    • Experiments
    • Feature flags
    • Funnels
    • Groups
    • Groups types
    • Insights
    • Invites
    • Members
    • Persons
    • Plugin configs
    • Plugins
    • Projects
    • Property definitions
    • Session recordings
    • Trends
    • Users
    • Data model
    • Overview
    • GDPR guidance
    • HIPAA guidance
    • CCPA guidance
    • Data egress & compliance
    • Overview
    • Code of conduct
    • Recognizing contributions
  • Using PostHog

  • Table of contents
      • Dashboards
      • Funnels
      • Group Analytics
      • Insights
      • Lifecycle
      • Path analysis
      • Retention
      • Stickiness
      • Trends
      • Heatmaps
      • Session Recording
      • Correlation Analysis
      • Experimentation
      • Feature Flags
      • Actions
      • Annotations
      • Cohorts
      • Data Management
      • Events
      • Persons
      • Sessions
      • UTM segmentation
      • Team collaboration
      • Organizations & projects
      • Settings
      • SSO & SAML
      • Toolbar
      • Notifications & alerts
  • Tutorials
    • All tutorials
    • Actions
    • Apps
    • Cohorts
    • Dashboards
    • Feature flags
    • Funnels
    • Heatmaps
    • Path analysis
    • Retention
    • Session recording
    • Trends
  • Support
  • Docs
  • Self-host
  • Runbook
  • Services
  • Kafka

Kafka

Last updated: Aug 09, 2022

On this page

  • Dictionary
  • Useful links

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

At PostHog we mainly use it to stream events from our ingestion pipeline to ClickHouse.

Dictionary

  • broker: a cluster is built by one or more servers. The servers forming the storage layer are called brokers
  • event: an event records the fact that "something happened" in the world or in your business. It is also called record or message in the documentation. When you read or write data to Kafka, you do this in the form of events. Conceptually, an event has a key, value, timestamp, and optional metadata headers
  • producers: client applications that publish (write) events to Kafka
  • consumer: client application subscribed to (read and process) events from Kafka
  • topic: group of events
  • partition: topics are partitioned, meaning a topic is spread over a number of "buckets" located on different Kafka brokers
  • replication: to make your data fault-tolerant and highly-available, every topic can be replicated, so that there are always multiple brokers that have a copy of the data just in case things go wrong

Useful links

  • Official documentation

Questions?

Was this page useful?

Next article

Resize disk

How-to List your pods Connect to the Kafka container to verify the data directory filesystem size (in this example 15GB) Resize the underlying PVC (in this example we are resizing it to to 20G) Note: while resizing the PVC you might get an error disk resize is only supported on Unattached disk, current disk state: Attached (see below for more details). In this specific case you need to temporary scale down the StatefulSet replica value to zero. This will briefly disrupt the Kafka service…

Read next article

Author

  • pjhul
    pjhul

Share

Jump to:

  • Dictionary
  • Useful links
  • Questions?
  • Edit this page
  • Raise an issue
  • Toggle content width
  • Toggle dark mode
  • About
  • Blog
  • Newsletter
  • Careers
  • Support
  • Contact sales

Product OS suite

Product overview

Analytics
  • Funnels
  • Trends
  • Paths

Pricing

Features
  • Session recording
  • Feature flags
  • Experimentation
  • Heatmaps

Customers

Platform
  • Correlation analysis
  • Collaboration
  • Apps

Community

Discussion
  • Questions?
  • Slack
  • Issues
  • Contact sales
Get involved
  • Roadmap
  • Contributors
  • Merch
  • PostHog FM
  • Marketplace

Docs

Getting started
  • PostHog Cloud
  • Self-hosted
  • Compare options
  • Tutorials
  • PostHog on GitHub
Install & integrate
  • Installation
  • Docs
  • API
  • Apps
User guides
  • Cohorts
  • Funnels
  • Sessions
  • Data
  • Events

Company

About
  • Our story
  • Team
  • Handbook
  • Investors
  • Careers
Resources
  • FAQ
  • Ask a question
  • Blog
  • Press
  • Merch
  • YouTube
© 2022 PostHog, Inc.
  • Code of conduct
  • Privacy
  • Terms