Releasing a new version

At the moment, we release a new version every month (unless it makes sense not to!). This might change in the future.

For consistency, releases happen on the last Monday of every month. Code freezes and break the release happen on the Wednesday before that. Each month there will be a different release owner in charge of the release, to be updated under this calendar.

If we've shipped features that we want to feature in the release notes, we use the label highlight on our pull request. If after the code freeze we have important bugfixes that we want to get into the release, we add the label release-[version]. This makes it easier for the release owner to figure out changes for the release blog post and to cherry-pick commits between the Code Freeze and the Release.

Version numbers

Every month we bump the minor in major.minor.patch. At the moment, we're at version 1 for major. This will only change once we have released sufficient functionality under stage 2 of our Roadmap.

Hopefully we will not have to do many patch versions, but if between versions we discover a breaking bug, we will.

Timeline

💡 For the context of this guide [version] is interpreted as the version of the release (e.g. 1.29.0).

Three business days before the release (Wednesday before the release), we institute a code freeze. Feel free to make an announcement on Slack before we cut the branch, so people can have a heads-up. Then, we branch master into release-[version] and deploy that to our playground environment, playground.posthog.com. We then host an hour-long "Break the release" session where everyone lends a hand in testing for any bugs. It's a recurring meeting, so you don't need to set it up.

Only bugfixes and finishing touches are allowed to be merged into this branch between the code freeze and the release going out. This gives us about three days to test the release.

The release manager is ultimately responsible for the timeline of the release. They are responsible for creating the "Code freeze" and "Break the release" calendar events as soon as possible. They should create these events under the Releases calendar linked up top.

Steps

Pre-release (Wednesday before the release)

1. Post in #dev about the upcoming release (replace <version> and <array draft pr> from Joe)

Release is happening next Monday. Which means 
1. There will be code freeze today (fixes that need to be cherry picked later should be tagged with `release-<version>`)
2. Please join the Break the Release meeting to help out testing on the Playground.
3. Shipped something awesome this month, please add a blurb or comment to <array draft pr> ([highlighted PRs](https://github.com/PostHog/posthog/pulls?q=is%3Apr+label%3A%22highlight+%3Astar%3A%22+)) :pray:

3. Start the release-[version] branch from master to initiate the code freeze.

4. Update the VERSION value in posthog/version.py and add an appropriate entry in posthog/versions.json. Then commit those changes:

   git checkout release-[version]
   git add posthog/version.py posthog/versions.json
   git commit -m "chore: Bump version to [version]"

5. Publish the release-[version] branch:

   git push -u origin release-[version]

   Note that this will result in a Docker image tagged release-[version]-unstable being built. It might take a while, but it should show up in Docker Hub within half an hour. You can check the build's status on the GitHub Actions page of the main repo.

   💡 Make sure you have doctl, helm, and k9s installed before going through the next steps. You can install all of these with brew install doctl helm k9s.

6. Upgrade PostHog playground

   1. The PostHog Playground uses a helm chart deployment on DigitalOcean. Find the playground cluster in our DigitalOcean Kubernetes clusters list.

   2. If this is your first time on DigitalOcean, you'll see the below screen. If it's not, or you don't see the Getting Started flow, click "Remind me how to use this file to connect to the cluster" in the "Config file" section under the "Overview" tab. Click Get Started.

      

   3. Copy the automatic connection script by clicking the copy icon.

      

   4. Open terminal and run the command you copied. This command will set the correct kubectl context for the playground environment. As a sanity check, run kubectl config current-context and make sure that the current context name has playground in it somewhere.

   5. Optional: Open another terminal window and run k9s. Use the arrow keys to scroll down to the PostHog clusters and keep an eye on this for the duration of the upgrade. k9s is a terminal GUI that makes it easier to manage and observe your deployed Kubernetes applications.

   6. Go to the playground.yaml file in the vpc repo and update the image: -> tag: value (release-[version]-unstable) with the new version. Commit the change as soon as release-[version]-unstable shows up in Docker Hub.

      ⚠️ Note that you might need to follow major upgrade notes as mentioned in the upgrade guide, the same way our users would be required to. If so, make any additional changes to the values.yaml file as needed.

   7. Copy the URL of the new playground.yaml file. You can get that by navigating to the file here, clicking Raw in the GitHub UI, and copying the URL of that page.

      

      

   8. In a separate terminal window, follow the upgrade instructions here. Replace values.yaml in the last upgrade command with the URL you copied in the previous step. Example:

      helm upgrade -f 'https://raw.githubusercontent.com/PostHog/vpc/main/client_values/posthog/playground.yaml?token=ABC' --timeout 30m --namespace posthog posthog posthog/posthog --atomic --wait --wait-for-jobs --debug

   9. Optional: Keep an eye on the progress of the upgrade in k9s

   10. If the helm upgrade command fails or if in the end the output for kubectl get pods -n posthog doesn't show everything as running, then ask team-platform for guidance.

   11. Optional: Verify playground is running the latest image by running kubectl get pod --namespace posthog. In the output of that command, you should see a row like posthog-web-6447ff5fdf-gs664. Copy this row (the numbers after posthog-web- will be different), and then run kubectl describe pod --namespace posthog posthog-web-6447ff5fdf-gs664. If you scroll up in that output, you should see a line like Image: posthog/posthog@sha256:daf43a4a4cd06658e41273bb8fe4a74f17b295d67c6f1e16c17243b5d09af7ee. This is the sha of the image that is running. You can compare this to the sha in Docker Hub to verify that the image is the latest.

   12. Go to the playground and test that everything is working as expected. Check that the version running is the same as the one we're releasing.

7. Time for the "Break the release" session! It's imperative that the session uses the published release-[version]-unstable image from Docker Hub to avoid any potential bugs creeping up in the final build stage. You're responsible for running the session, prepare the release checklist doc by adding the template at the top. Note that you're also responsible for making sure everything is tested and for cherry picking the fixes and prs tagged with release-<version> into the release branch.

8. Figure out what's updated in this release with the command below or by asking the Product or Engineering Team. The command will output the entire commit list to changelog.txt, sorted by PR type and scope. You can use this list to obtain external contributions to highlight in the Array. In addition, you can look for the highlight tag in PRs but be mindful it's not used very consistently.

   git checkout release-[version]
   git log --pretty=format:"%s %ae" origin/release-[old-version]..head | sort -t ':' -k 1,1 -s > changelog.txt

9. Write up the PostHog Array blog post. Please tag Joe Martin for review, as this helps Marketing coordinate other announcements. Do not release the post until the day of release.

10. Share the PostHog Array blog post with all partners listed in the PostHog Marketplace via the dedicated Slack channels. Don't have access to them all? Please ask Joe Martin to do this instead.

Launch (day of the release)

1. Tag the version in GitHub. This will also build and push the release-[version], latest-release (for both PostHog base & FOSS) Docker images to Docker Hub. Please do this once the release branch is finalized, some users may see the image on Docker Hub and update immediately.
   git tag -a [version] -m "Version [version]"
   git push --follow-tags

2. Create a new charts-clickhouse branch named bump-[version] to update the Helm chart:
   1. In Chart.yaml update appVersion to the new version.
   2. In values.yaml update image.default to point to the new tag (i.e. :release-[version]).
   3. In ALL_VALUES.md update the default value of image.default to what you set in the previous step. Also, update the AppVersion shields.io badge at the top.
   4. Push the relevant changes and create a PR. Add the bump minor label to the PR. Do not merge until the release-[version] branch is built. (You can see that in Docker Hub)
3. Publish the PostHog Array blog post.
4. Create a new main repo (posthog) branch named sync-[version]. Cherry-pick the release-[version] commits updating version.py and versions.json into sync-[version] and create a PR to get them into master. Merging this to master will notify users that an update is available. The Array post should be out at this point so that the "Release notes" link isn't a 404.
5. Go to the EWXT9O7BVDC2O CloudFront distribution to the "Invalidations" tab and add a new one with /* value. This will refresh the CloudFront cache so that users can see the new version.
6. Send a message on the PostHog Users Slack (community) in #announcements to let everyone know the release has shipped.
7. Send the newsletter with the PostHog Array. The Marketing Team will arrange this, provided Joe Martin has been tagged for review in the PostHog Array blog post.