We've decided to make less money: We've slashed our pricing for session replay. They're now more than 50% cheaper for most customers.
Pipeline Team


What we're building

  • PostHog Customer Data Platform

    We cover some of the functionality of existing CDP solutions, such as Segment. By creating a UI which specifically encompasses the ideas of "Sources" and "Destinations", along with building out more of the integrations, we can turn PostHog into a leading CDP solution.

    Project updates

    No updates yet. Engineers are currently hard at work, so check back soon!


Here’s what we’re considering building next. Vote for your favorites or share a new idea on GitHub.

Recently shipped

New data pipelines UI

Back in May we launched the beta for a new data pipelines UI, and today we've rolled that beta out to everyone and retired the old version.

As before, the new UI breaks the data pipeline down into destinations and transformations, filters for your active pipelines, and includes teeny, weeny graphs inside so you can get volumes at a glance.


Goals: Q3 2024

  1. Test Warpstream as PoC and decide whether to do it or not
  2. Pipeline scalability Improving pipeline throughput
  3. Help other teams ship fast
  4. Stretch: better e2e monitoring



Our team owns our ingestion pipeline until Kafka in front of ClickHouse. That means we own:

  • Capture APIs
  • Ingestion app server
  • Data pipelines, including:
    • Transformation apps
    • Data Export and webhooks
  • Client libraries, where it pertains to event ingestion
  • Kafka setup, where it pertains to event ingestion

Our work generally falls into one of three categories, in order of priority:

Ingestion robustness

On the road to providing the best events pipeline in the world, we need to build a system that is robust.

To do so, we must ensure, in order of priority:

  • Data integrity: Events ingested should be correct
  • Availability: We should not lose events
  • Scalability: We should be able to scale to massive event volumes
  • Maintainability: It should be easy to debug and contribute to our ingestion pipeline

Thus, it is our responsibility to consistently revise our past decisions and improve processes where we see fit, from client library behaviors to ClickHouse schemas.

Scaffolding to support core PostHog features

In order to achieve company goals or introduce new features (often owned by other teams), changes to our ingestion pipeline may be required.

An example of this is the work to remodel our events to store person and group data, which is essential to ensuring we can provide fast querying for users. While querying data is not owned by this team, the change to enable faster queries requires a large restructuring of our events pipeline, and thus we are owners of that component of the project.

In short, a core responsibility of our team is to enable other teams to be successful.


We're building a flexible and integrated data pipeline, which we treat as a separate product just like product analytics and feature flags, for example. The PostHog data warehouse is one destination, and the most important, but we care about getting data into all the places where it is valuable.

How do we work?

We run a 30 minute sync meeting on Monday, Wednesdays, and Fridays, and extend the slot if we feel the need to have a longer synchronous discussion about a specific topic. We look at the Pipeline Team GitHub Board, and we document every sync in this doc.

We are happy to sync anytime if we feel it is important to do so. This is generally coordinated on Slack where someone will start a Slack huddle. Some of the reasons we sync include: debugging outages, sharing context (including shadowing), making decisions when there's been a deadlock, and pairing sessions.

We have a single owner assigned to each priority and sprint goal, but we work together on goals as a team. We try to avoid lone-wolfing, so in general we'll have 3 sprint goals per sprint.

Secondary (a.k.a. Luigi)

We have a secondary schedule. The idea is to have a single person catch all of the interruptions and context switching. A secondary rotation is two weeks long and aligns with our sprints. We don't assign sprint goals to the secondary.

During this rotation their first priority are these responsibilities:

  • Firefighting
  • Questions in #support-pipeline
  • Customer Success triage
  • Adding or improving runbooks
  • Follow ups from outages
  • Cleaning up Sentry

If there is a long running customer issue, we'll assign an owner as a team.

It's acceptable for the secondary responsibilities to take up all of your time. Otherwise, you can help out the rest of the team with sprint goals. It's up to you to prioritize correctly.

Slack channel