❗️ This guide is intended only for development of PostHog itself. If you're looking to deploy PostHog for your product analytics needs, go to Self-host PostHog.
What does PostHog look like on the inside?
Before jumping into setup, let's dissect a PostHog.
The app itself is made up of 4 components that run simultaneously:
- Django server
- Celery worker (handles execution of background tasks)
- Node.js plugin server (handles event ingestion and apps/plugins)
- React frontend built with Node.js
These components rely on a few external services:
- ClickHouse – for storing big data (events, persons – analytics queries)
- Kafka – for queuing events for ingestion
- MinIO – for storing files (session recordings, file exports)
- PostgreSQL – for storing ordinary data (users, projects, saved insights)
- Redis – for caching and inter-service communication
- Zookeeper – for coordinating Kafka and ClickHouse clusters
When spinning up an instance of PostHog for development, we recommend the following configuration:
- the external services run in Docker over
- PostHog itself runs on the host (your system)
This is what we'll be using in the guide below.
It is also technically possible to run PostHog in Docker completely, but syncing changes is then much slower, and for development you need PostHog dependencies installed on the host anyway (such as formatting or typechecking tools). The other way around – everything on the host, is not practical due to significant complexities involved in instantiating Kafka or ClickHouse from scratch.
The instructions here assume you're running macOS or the current Ubuntu Linux LTS (22.04).
For other Linux distros, adjust the steps as needed (e.g. use
pacman in place of
Windows isn't supported natively. But, Windows users can run a Linux virtual machine. The latest Ubuntu LTS Desktop is recommended. (Ubuntu Server is not recommended as debugging the frontend will require a browser that can access localhost.)
In case some steps here have fallen out of date, please tell us about it – feel free to submit a patch!
Install Xcode Command Line Tools if you haven't already:
Install the package manager Homebrew by following the instructions here.
After installation, make sure to follow the instructions printed in your terminal to add Homebrew to your
$PATH. Otherwise the command line will not know about packages installed with
- Install Docker Desktop and in its settings give Docker at least 4 GB of RAM (or 6 GB if you can afford it) and at least 4 CPU cores.
Install Docker following the official instructions here.
build-essentialpackage:Terminalsudo apt install -y build-essential
Common prerequisites for both macOS & Linux
127.0.0.1 kafka clickhouseto
/etc/hosts. You can do it in one line with:Terminalecho '127.0.0.1 kafka clickhouse' | sudo tee -a /etc/hosts
ClickHouse and Kafka won't be able to talk to each other without these mapped hosts.
Clone the PostHog repository. All future commands assume you're inside the
posthog/folder.Terminalgit clone https://github.com/PostHog/posthog && cd posthog/
Get things up and running
1. Spin up external services
In this step we will start all the external services needed by PostHog to work.
We'll be using
docker compose, which is the successor to
docker-compose. One of its features is better compatibility with ARM environments like Apple Silicon Macs. (See Docker documentation for details.)
docker compose -f docker-compose.dev.yml up
Friendly tip 1: If you see
Error while fetching server API version: 500 Server Error for http+docker://localhost/version:, it's likely that Docker Engine isn't running.
Friendly tip 2: If you see "Exit Code 137" anywhere, it means that the container has run out of memory. In this case you need to allocate more RAM in Docker Desktop settings.
Friendly tip 3: You might need
sudo– see Docker docs on managing Docker as a non-root user.
Friendly tip 4: If you see
Error: (HTTP code 500) server error - Ports are not available: exposing port TCP 0.0.0.0:5432 -> 0.0.0.0:0: listen tcp 0.0.0.0:5432: bind: address already in use, - refer to this stackoverflow answer. In most cases, you can solve this by stopping the
sudo service postgresql stop
Second, verify via
docker ps and
docker logs (or via the Docker Desktop dashboard) that all these services are up and running. They should display something like this in their logs:
# docker ps NAMESCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES567a4bb735be clickhouse/clickhouse-server:22.3 "/entrypoint.sh" 3 minutes ago Up 24 seconds 0.0.0.0:8123->8123/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:9009->9009/tcp, 0.0.0.0:9440->9440/tcp posthog-clickhouse-19dc22c70865d bitnami/kafka:2.8.1-debian-10-r99 "/opt/bitnami/script…" 3 minutes ago Up 24 seconds 0.0.0.0:9092->9092/tcp posthog-kafka-1add6475ae0db postgres:12-alpine "docker-entrypoint.s…" 3 minutes ago Up 24 seconds 0.0.0.0:5432->5432/tcp posthog-db-16037fb28659b minio/minio "sh -c 'mkdir -p /da…" 3 minutes ago Up 24 seconds 9000/tcp, 0.0.0.0:19000-19001->19000-19001/tcp posthog-object_storage-1d80a9304f4a7 zookeeper:3.7.0 "/docker-entrypoint.…" 3 minutes ago Up 24 seconds 2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp posthog-zookeeper-1d2d00eae3fc0 redis:6.2.7-alpine "docker-entrypoint.s…" 3 minutes ago Up 24 seconds 0.0.0.0:6379->6379/tcp posthog-redis-1 posthog-redis-1# docker logs posthog-db-1 -n 12021-12-06 13:47:08.325 UTC  LOG: database system is ready to accept connections# docker logs posthog-redis-1 -n 11:M 06 Dec 2021 13:47:08.435 * Ready to accept connections# docker logs posthog-clickhouse-1 -n 1Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.# docker logs posthog-kafka-1[2021-12-06 13:47:23,814] INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)# docker logs posthog-zookeeper-1# Because ClickHouse and Kafka connect to Zookeeper, there will be a lot of noise here. That's good.
Friendly tip: Kafka is currently the only x86 container used, and might segfault randomly when running on ARM. Restart it when that happens.
Finally, install Postgres locally. Even if you are planning to run Postgres inside Docker, we need a local copy of Postgres (version 11+) for its CLI tools and development libraries/headers. These are required by
pip to install
- On macOS:Terminalbrew install postgresql
This installs both the Postgres server and its tools. DO NOT start the server after running this.
- On Debian-based Linux:Terminalsudo apt install -y postgresql-client postgresql-contrib libpq-dev
This intentionally only installs the Postgres client and drivers, and not the server. If you wish to install the server, or have it installed already, you will want to stop it, because the TCP port it uses conflicts with the one used by the Postgres Docker container. On Linux, this can be done with
sudo systemctl disable postgresql.service.
On Linux you often have separate packages:
postgres for the tools,
postgres-server for the server, and
libpostgres-dev for the
psycopg2 dependencies. Consult your distro's list for an up-to-date list of packages.
2. Prepare the frontend
- Install nvm, with
brew install nvmor by following the instructions at https://github.com/nvm-sh/nvm. If using fish, you may instead prefer https://github.com/jorgebucaran/nvm.fish.
After installation, make sure to follow the instructions printed in your terminal to add NVM to your
$PATH. Otherwise the command line will use your system Node.js version instead.
Install the latest Node.js 16 (the version used by PostHog in production) with
nvm install 16. You can start using it in the current shell with
nvm use 16.
Install yarn with
npm install -g yarn@1.
Install Node packages by running
yarn typegen:writeto generate types for Kea state management logics used all over the frontend.
The first time you run typegen, it may get stuck in a loop. If so, cancel the process (
Ctrl+C), discard all changes in the working directory (
git reset --hard), and run
yarn typegen:writeagain. You may need to discard all changes once more when the second round of type generation completes.
3. Prepare plugin server
Assuming Node.js is installed, run
yarn --cwd plugin-server to install all required packages. You'll also need to install the
brotli compression library:
- On macOS:Terminalbrew install brotli
- On Debian-based Linux:Terminalsudo apt install -y brotli
We'll run the plugin server in a later step.
4. Prepare the Django server
Install a few dependencies for SAML to work. If you're on macOS, run the command below, otherwise check the official xmlsec repo for more details.
- On macOS:Terminalbrew install libxml2 libxmlsec1 pkg-config
- On Debian-based Linux:Terminalsudo apt install -y libxml2 libxmlsec1-dev pkg-config
- On macOS:
Install Python 3.8.
On macOS, you can do so with Homebrew:
brew install firstname.lastname@example.org.
On Debian-based Linux:Terminalsudo add-apt-repository ppa:deadsnakes/ppa -ysudo apt updatesudo apt install python3.8 python3.8-venv python3.8-dev -y
Make sure when outside of
venv to always use
python3 instead of
python, as the latter may point to Python 2.x on some systems. If installing multiple versions of Python 3, such as by using the
deadsnakes PPA, use
python3.8 instead of
You can also use pyenv if you wish to manage multiple versions of Python 3 on the same machine.
Create the virtual environment in current directory called 'env':Terminalpython3 -m venv env
Activate the virtual environment:Terminal# For bash/zsh/etc.source env/bin/activate# For fishsource env/bin/activate.fish
Upgrade pip to the latest version:Terminalpip install -U pip
Install requirements with pip
If your workstation is an Apple Silicon Mac, the first time your run
pip installyou must set custom OpenSSL headers:Terminalbrew install opensslCFLAGS="-I /opt/homebrew/opt/openssl/include $(python3.8-config --includes)" LDFLAGS="-L /opt/homebrew/opt/openssl/lib" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt
These will be used when installing
psycopg2. After doing this once, and assuming nothing changed with these two packages, next time simply run:Terminalpip install -r requirements.txt
If on an x86 platform, simply run the latter version.
Install dev requirementsTerminalpip install -r requirements-dev.txt
5. Prepare databases
We now have the backend ready, and Postgres and ClickHouse running – these databases are blank slates at the moment however, so we need to run migrations to e.g. create all the tables:
Friendly tip: The error
fe_sendauth: no password suppliedconnecting to Postgres happens when the database is set up with a password and the user:pass isn't specified in
Another friendly tip: You may run into
psycopg2errors while migrating on an ARM machine. Try out the steps in this comment to resolve this.
6. Start PostHog
Now start all of PostHog (backend, worker, plugin server, and frontend – simultaneously) with:
Open http://localhost:8000 to see the app.
Note: The first time you run this command you might get an error that says "layout.html is not defined". Make sure you wait until the frontend is finished compiling and try again.
To get some practical test data into your brand-new instance of PostHog, run
DEBUG=1 ./manage.py generate_demo_data. For a list of useful arguments of the command, run
DEBUG=1 ./manage.py generate_demo_data --help.
This is it! You can now change PostHog in any way you want. See Project Structure for an intro to the repository's contents.
To commit changes, create a new branch based on
master for your intended change, and develop away. Just make sure not use to use
release-* patterns in your branches unless putting out a new version of PostHog, as such branches have special handling related to releases.
For a PostHog PR to be merged, all tests must be green, and ideally you should be introducing new ones as well – that's why you must be able to run tests with ease.
For backend, simply use:
You can narrow the run down to only files under matching paths:
Or to only test cases with matching function names:
pytest posthog/test/test_entity_model.py -k test_inclusion
To see debug logs (such as ClickHouse queries), add argument
For Cypress end-to-end test, run
bin/e2e-test-runner. This will temporarily install required dependencies inside the project, spin up a test instance of PostHog, and show you the Cypress interface, from which you'll manually choose tests to run.
Once you're done, terminate the command with cmd + C. Be sure to wait until the command terminates gracefully so temporary dependencies are removed from
package.json and you don't commit them accidentally.
For frontend tests, all you need is
Extra: Working with feature flags
When developing locally with environment variable
DEBUG=1 (which enables a setting called
all analytics inside your local PostHog instance is based on that instance itself – more specifically, the currently selected project.
This means that your activity is immediately reflected in the current project, which is potentially useful for testing features
– for example, which feature flags are currently enabled for your development instance is decided by the project you have open at the very same time.
So, when working with a feature based on feature flag
foo-bar, add a feature flag with this key to your local instance and release it there.
If you'd like to have ALL feature flags that exist in PostHog at your disposal right away, run
python3 manage.py sync_feature_flags – they will be added to each project in the instance, fully rolled out by default.
This command automatically turns any feature flag ending in
_EXPERIMENT as a multivariate flag with
Extra: Debugging the backend in PyCharm
With PyCharm's built in support for Django, it's fairly easy to setup debugging in the backend. This is especially useful when you want to trace and debug a network request made from the client all the way back to the server. You can set breakpoints and step through code to see exactly what the backend is doing with your request.
- Setup Django configuration as per JetBrain's docs.
- Click Edit Configuration to edit the Django Server configuration you just created.
- Point PyCharm to the project root (
posthog/) and settings (
- Add these environment variables