Linking Amazon S3 as a source

The Amazon S3 source imports CSV, JSON Lines, and Parquet files, and Delta Lake tables, from an S3 bucket into PostHog. PostHog keeps the tables up to date on the schedule you choose, so you can query your files alongside your product data.

Prerequisites

You need:

• An S3 bucket that contains your files.
• An AWS access key with read access to the bucket.

Create an access key

1. Open IAM in the AWS console.
2. Under Access management, select Policies and click Create policy.
3. Select the JSON tab and paste this policy. Replace your-bucket with the name of your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::your-bucket"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::your-bucket/*"
    }
  ]
}

4. Click Next, give the policy a name, and click Create policy.
5. Under Access management, select Users and click Create user. Enter a user name and click Next.
6. Select Attach policies directly, select the policy that you created, and click Next. Then click Create user.
7. Open the user, select the Security credentials tab, and click Create access key.
8. Select Application running outside AWS and click Next. Then click Create access key.
9. Copy the Access key ID and the Secret access key. AWS shows the secret only once.

Tip: If you don't have data in S3 yet, you can use an ETL tool such as Airbyte, Fivetran, or Stitch to export data to the bucket in Parquet format.

Adding a data source

1. In PostHog, go to the Sources tab of the data pipeline section.
2. Click + New source and click Link next to this source.
3. Enter your credentials (see Configuration below) and click Next.
4. Select the tables you want to sync, choose a sync method and frequency, then click Import.

Once the syncs are complete, you can start querying this data in PostHog.

When you link S3, enter:

• Files URL: The URL of your files. You can paste an s3:// URI, such as s3://your-bucket/orders/, or an object URL from the AWS console, such as https://your-bucket.s3.us-east-1.amazonaws.com/orders/**/*.parquet.
• Access key ID and Secret access key: The access key that you created.

Files and tables

Files URL

The Files URL tells PostHog which files to import. It starts with the bucket (or container) and can point to a folder, a single file, or a pattern:

• Use * to match any part of a file or folder name, such as orders/*.parquet.
• Use ** to match across folders, such as orders/**/*.parquet.
• A URL without a wildcard imports the folder or file it points to.

The bucket name must be exact. Put wildcards only in the path after the bucket.

Supported file formats

PostHog also reads gzip-compressed files, such as .csv.gz.

By default, PostHog detects the format from the file extension. If your files have no extension or an unusual one, set File format to the format of your files. For CSV files, you can also tell PostHog if double quotes enclose the values. If they don't, PostHog reads quote characters as plain text.

How files become tables

PostHog finds the files that match the URL and creates:

• One table for each folder.
• One table for each file directly in the folder that the URL points to.
• One table for each Delta Lake table.

If all the files share the same columns, such as a daily export, turn on Combine every file into one table? and enter a table name. PostHog then imports all the files into that one table. Delta Lake tables still get one table each.

After you connect, select the tables to import, then choose a sync method and a sync frequency for each table.

Limits

• A source can have up to 1,000 tables.
• A table can have up to 50,000 files. If a table has more files, the sync fails. Use a more specific URL, or split the files into more folders.

Sync modes

Each table supports three sync methods:

Use Incremental for most tables. Use Append only for folders where files are only added, such as event logs. See sync methods for more about each method.

Rows that PostHog imports count toward your data warehouse rows synced.

Configuration

Supported tables

The tables available from this source are discovered from your account when you connect it, so the exact list depends on your data. Once connected, you can pick which tables to sync from the sources tab.

Troubleshooting

• The storage settings are invalid: Make sure that the files URL is in the format that the example shows, and that the bucket name doesn't contain a wildcard.
• The storage provider denied access to the bucket: Make sure that the credentials are correct and can list and read the files in the bucket.
• The bucket doesn't exist: Make sure that the bucket name in the URL is correct.
• No importable files were found: Make sure that the URL matches at least one file in a supported format. If your files have no extension, set File format to the format of your files.
• The table has more files than PostHog can import: Use a more specific URL, or split the files into more folders.
• Couldn't parse the file: Make sure that File format matches your files. For CSV files, try to change the double quotes setting. For JSON files, make sure each line holds one JSON object, not a single JSON array.
• The Delta table uses features PostHog can't read: Some Delta Lake features aren't supported. Export the table to Parquet files and import those files instead.

• The bucket is in a different region: Use a files URL that includes the region of the bucket, such as https://your-bucket.s3.eu-central-1.amazonaws.com/orders/.
• Files in archive storage classes: PostHog can't read files in the S3 Glacier storage classes or in the archive tiers of S3 Intelligent-Tiering. Restore the files to a standard storage class, or use a files URL that doesn't match the archived files.

If your sync is failing or data looks wrong, see the Data warehouse troubleshooting guide. If that doesn't help, contact support – we're happy to help.