> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

# S3 destination for batch exports

Canonical URL: https://posthog.com/docs/data-pipelines/batch-exports/s3

> **Cloud region:** This page uses US Cloud hosts. For an EU Cloud project, use `https://eu.posthog.com` instead of `https://app.posthog.com` (app host for links).

## On this page

- [Models](#models)
- [Creating the batch export](#creating-the-batch-export)
- [S3 configuration](#s3-configuration)
- [AWS S3 authentication](#aws-s3-authentication)

Batch exports support two S3-family destinations: **AWS S3** for buckets hosted on Amazon Web Services, and **S3-compatible** for any other S3-compatible object storage provider such as Cloudflare R2, Google Cloud Storage, MinIO, or DigitalOcean Spaces.

**New integration types for S3 destinations**

AWS S3 and S3-compatible connections are now backed by reusable integrations, which store your credentials securely and let you reuse them across multiple exports. This is the preferred way to set up new exports going forward.

Existing exports will be migrated to use integrations automatically in the near future.

## Models

This section describes the models that can be exported to S3.

> **Note:** New fields may be added to these models over time. Therefore, it is recommended that any downstream processes are able to handle additional fields being added to the exported files.

### Events model

This is the default model for S3 batch exports. When exported in the Parquet file format, the schema is:

| Field | Type | Description |
| --- | --- | --- |
| uuid | `STRING` | The unique ID of the event within PostHog |
| event | `STRING` | The name of the event that was sent |
| distinct\_id | `STRING` | The `distinct_id` of the user who sent the event |
| person\_id | `STRING` | The ID of the person at the time the event was ingested. Rows are never updated regardless of underlying person changes |
| properties | `STRING` | All the properties sent along with the event, stored as a JSON-encoded string in Parquet |
| person\_properties | `STRING` | All the person properties at the time the event was ingested. Later changes to the person do not update exported rows. Stored as a JSON-encoded string in Parquet |
| elements\_chain | `STRING` | The chain of DOM elements for `$autocapture` events. Empty for other events |
| timestamp | `TIMESTAMP` | When the event occurred, as reported by the client. Microsecond precision, UTC |
| created\_at | `TIMESTAMP` | When PostHog ingested the event. Microsecond precision, UTC |
| \_inserted\_at | `TIMESTAMP` | Internal field used by batch exports to track export progress. Included in files but safe to ignore |

> **Note:** The types above describe the **Parquet** file format. Other file formats represent the same data differently. For example, in **JSONLines** exports the timestamp fields (`timestamp`, `created_at`, and `_inserted_at`) are ISO 8601 strings, and `properties` and `person_properties` are nested JSON objects rather than JSON-encoded strings.

### Persons model

The schema of the persons model when exported in the Parquet file format is:

| Field | Type | Description |
| --- | --- | --- |
| team\_id | `BIGINT` | The ID of the project (team) the person belongs to |
| distinct\_id | `STRING` | A `distinct_id` associated with the person |
| person\_id | `STRING` | The ID of the person for this (`team_id`, `distinct_id`) pair |
| properties | `STRING` | A JSON object with the latest person properties, stored as a JSON-encoded string in Parquet |
| person\_distinct\_id\_version | `BIGINT` | Internal version of the person-to-`distinct_id` mapping, used by batch exports during merges |
| person\_version | `BIGINT` | Internal version of the person's properties, used by batch exports during merges |
| created\_at | `TIMESTAMP` | When the person was created. Microsecond precision, UTC |
| \_inserted\_at | `TIMESTAMP` | Internal field used by batch exports to track export progress. Included in files but safe to ignore |
| is\_deleted | `BOOLEAN` | Whether the person has been deleted |

Each export contains one row per `(team_id, distinct_id)` pair, mapped to their corresponding `person_id` and latest `properties`.

> **Note:** The persons model only includes persons that have a person profile in PostHog. If your project has [person profile processing disabled](https://posthog.com/docs/data/anonymous-vs-identified-events.md) (via `person_profiles: 'identified_only'`, `person_profiles: 'never'`, or by sending events with `$process_person_profile: false`), anonymous users who have never been identified will **not** appear in the persons export. To count unique users including those without person profiles, you can fall back to `distinct_id` from the events model. See the example queries in each destination's documentation for details.

> **Note:** As with the events model, these types describe the **Parquet** file format. In **JSONLines** exports, `created_at` and `_inserted_at` are ISO 8601 strings and `properties` is a nested JSON object rather than a JSON-encoded string.

### Sessions model

You can view the schema for the sessions model in the configuration form when creating a batch export (there are a few too many fields to display here!).

## Creating the batch export

1.  Go to [Data > Destinations](https://app.posthog.com/data-management/destinations) in the left sidebar.
2.  Click **\+ New destination** in the top-right corner.
3.  Choose the destination that matches your storage provider:
    -   **AWS S3** – Select this if you're exporting to an Amazon S3 bucket.
    -   **S3-compatible** – Select this for any non-AWS S3-compatible provider (Cloudflare R2, Google Cloud Storage, MinIO, DigitalOcean Spaces, etc.).
4.  Click the **\+ Create** button.
5.  Set up a connection by clicking **Configure new AWS S3 connection** (or **Configure new S3-compatible connection** for non-AWS providers) in the **Connection** dropdown. This opens a setup modal where you choose an [authentication method](#aws-s3-authentication), provide a name, and enter your credentials. Connections are stored securely and can be reused across multiple exports.
6.  Fill in the remaining [configuration details](#s3-configuration).
7.  Click **Create** to finalize.
8.  Done! The batch export will schedule its first run on the start of the next period.

## S3 configuration

Both S3-family destinations share a common set of configuration fields. Some fields are specific to one destination type.

### Common fields

-   **Bucket name:** The name of the bucket where data is exported.
-   **Region:** The region where the bucket is located.
-   **Key prefix:** A key prefix for each object created. Supports [template variables](#s3-key-prefix-template-variables).
-   **Format:** The file format for the export. See [S3 file formats](#s3-file-formats).
-   **Max file size (MiB):** Split exported data into multiple files when it exceeds this size. Set to 0 or leave empty for no splitting.
-   **Compression:** A compression method (like gzip or zstd) for exported files, or no compression.
-   **Connection:** A reusable integration that stores your credentials. For AWS S3 destinations, you can authenticate using [IAM role assumption](#iam-role-recommended) or [access keys](#access-keys). For S3-compatible destinations, access keys are required. Click **Configure new AWS S3 connection** (or **Configure new S3-compatible connection**) to create one.
-   **Events to exclude:** A list of events to omit from the exported data.

### AWS S3-only fields

-   **Encryption:** Server-side encryption method (`AES256` or `aws:kms`) for AWS to encrypt data at rest.
-   **AWS KMS Key ID:** The KMS Key ID to use for server-side encryption. Only required when encryption is set to `aws:kms`.

### S3-compatible-only fields

-   **Endpoint URL:** The endpoint URL for your storage provider (e.g., `https://<ACCOUNT_ID>.r2.cloudflarestorage.com` for Cloudflare R2 or `https://storage.googleapis.com` for GCS). This is set when you create the connection, and can't be changed from the export's configuration form afterwards.
-   **Virtual style addressing:** Enable this if your provider requires virtual hosted-style bucket addressing. Check your provider's documentation – leave unchecked if unsure.

### S3 key prefix template variables

The key prefix provided for data exporting can include template variables which are formatted at runtime. All template variables are defined between curly brackets (for example `{day}`). This allows you partition files in your S3 bucket, such as by date.

Template variables include:

-   Date and time variables:
    -   `year`.
    -   `month`.
    -   `day`.
    -   `hour`.
    -   `minute`.
    -   `second`.
-   Name of the table exported (for example, 'events' or 'persons')
    -   `table`.
-   Batch export data bounds:
    -   `data_interval_start`.
    -   `data_interval_end`.

So, as an example, setting `{year}-{month}-{day}_{table}/` as a key prefix, will produce files prefixed with keys like `2023-07-28_events/`.

### S3 file formats

PostHog S3 batch exports support two file formats for exporting data:

-   [JSON lines](https://jsonlines.org/).
-   [Apache Parquet](https://parquet.apache.org/) (latest version of the format specification is the only one supported).

The batch export format is selected via a drop down menu when creating or editing an export.

We intend to add support for other common formats, and format-specific configuration options. You can follow the [roadmap](https://github.com/PostHog/posthog/issues/15997) to track progress.

### Compression

Each file format supports a variety of compression methods. The compression method you choose can have a significant effect on the exported file size and the overall time taken to export the data. From our own internal testing, we would recommend using Parquet with zstd compression for the best combination of speed and file size.

> **Note on Parquet compression:** The compression type is included in the file extension, even for Parquet files. For example, files compressed with zstd will have the extension `parquet.zst`. Since compression is embedded in the format itself, the file should be read directly as a Parquet file and not uncompressed first.

### Manifest file

If you specify a max file size in your configuration, several files may be exported. In order to know when the export is complete, we send a `manifest.json` file (with the same prefix as the other files) once all the data files have been exported. This manifest file contains the key names of all the files exported.

### S3-compatible storage providers

When creating a batch export, select the **S3-compatible** destination for any non-AWS S3-compatible storage provider. Below are configuration tips for providers we have tested.

#### MinIO

-   Set the **Endpoint URL** to your MinIO instance's host and port, for example: `https://my-minio-storage:9000`.

#### Cloudflare R2

-   Set the **Endpoint URL** to the following after replacing your account ID: `https://<ACCOUNT_ID>.r2.cloudflarestorage.com`.
-   From the **Region** dropdown, select one of the Cloudflare R2 regions that correspond to your bucket, like "Cloudflare R2 — Automatic (AUTO)".

#### Google Cloud Storage (GCS)

Access to GCS for batch exports follows a similar process to accessing [BigQuery](https://posthog.com/docs/data-pipelines/batch-exports/bigquery.md) as a Service Account is required:

1.  Follow the steps in the [BigQuery batch export documentation](https://posthog.com/docs/data-pipelines/batch-exports/bigquery.md) to create a Service Account.
2.  Create a [HMAC key for your Service Account](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#console).
3.  Grant the Service Account the `Storage Object User` role or a custom role with at least the following permissions:
    -   `storage.multipartUploads.abort`
    -   `storage.multipartUploads.create`
    -   `storage.multipartUploads.list`
    -   `storage.multipartUploads.listParts`
    -   `storage.objects.create`
    -   `storage.objects.delete`
4.  Use the HMAC key access key and secret key as **Access Key ID** and **Secret Access Key** respectively when configuring your batch export.
5.  Set the **Endpoint URL** to: `https://storage.googleapis.com`.
6.  Select the appropriate GCP region from the **Region** dropdown.

## AWS S3 authentication

When creating an AWS S3 connection, you choose between two authentication methods: **IAM role assumption** (recommended) or **access keys**.

### IAM role (recommended)

With IAM role assumption, PostHog uses AWS STS to assume a role in your AWS account and obtain short-lived credentials. This avoids storing long-lived access keys.

To set up IAM role authentication:

1.  In the AWS S3 connection modal, select the **Assume IAM role** tab.
2.  Create an IAM role in your AWS account with a trust policy that allows PostHog's role to assume it. The PostHog role ARN depends on your cloud region:
    -   **US Cloud** - `arn:aws:iam::309986977637:role/posthog-external-batch-exports`
    -   **EU Cloud** - `arn:aws:iam::623789312881:role/posthog-external-batch-exports`
3.  Add an `sts:ExternalId` condition to the trust policy. Set the value to `posthog-<your_organization_id>` (your organization ID is shown in the setup modal). PostHog verifies this condition is enforced and exports fail without it.
4.  Attach a permissions policy to the role that grants `s3:PutObject` and `s3:AbortMultipartUpload` on your destination bucket and prefix. If the bucket uses KMS encryption, also grant `kms:GenerateDataKey` and `kms:Decrypt` on the key.
5.  Paste the role ARN (e.g., `arn:aws:iam::123456789012:role/posthog-batch-exports`) into the **IAM role ARN** field in the setup modal.

Here's an example trust policy:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::309986977637:role/posthog-external-batch-exports"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "posthog-<your_organization_id>"
        }
      }
    }
  ]
}
```

And an example permissions policy:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}
```

> **Note:** Self-hosted users should check with their instance administrator for the correct PostHog role ARN to trust.

### Access keys

With access keys, you provide an AWS Access Key ID and Secret Access Key that PostHog stores encrypted. This method works but uses long-lived credentials. Prefer IAM role assumption when possible.

To set up access key authentication:

1.  In the AWS S3 connection modal, select the **Access keys** tab.
2.  Enter your **AWS Access Key ID** and **AWS Secret Access Key**.
3.  The IAM user associated with these keys needs `s3:PutObject` and `s3:AbortMultipartUpload` permissions on your destination bucket.