> AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

# Linking AWS Glue Data Catalog as a source

Canonical URL: https://posthog.com/docs/data-pipelines/sources/aws-glue-data-catalog

> **Cloud region:** This page uses US Cloud hosts. For an EU Cloud project, use `https://eu.posthog.com` instead of `https://us.posthog.com` or `https://app.posthog.com` (app host for links).

## On this page

- [Prerequisites](#prerequisites)
- [Adding a data source](#adding-a-data-source)
- [Creating IAM credentials](#creating-iam-credentials)
- [Sync modes](#sync-modes)
- [Configuration](#configuration)
- [Supported tables](#supported-tables)
- [Troubleshooting](#troubleshooting)

**Beta release**

This source is currently in **beta**. It works end to end, but you may still hit rough edges. [Let us know](https://us.posthog.com/#panel=support%3Asupport%3Adata_warehouse%3A%3Atrue) if you do.

![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_9608fcca70)

![](https://res.cloudinary.com/dmukukwp6/image/upload/texture_tan_dark_a92b0e022d)

Let AI connect your sources for you

Skip the manual setup — run this in your project and the wizard auto-detects your databases and APIs and connects them to PostHog.

`npx @posthog/wizard warehouse`

![PostHog Wizard hedgehog](https://res.cloudinary.com/dmukukwp6/image/upload/wizard_3f8bb7a240.png)

![](https://res.cloudinary.com/dmukukwp6/image/upload/wizard_3f8bb7a240.png)Let AI connect your sources for you

The AWS Glue Data Catalog connector syncs your Glue metadata, jobs, crawlers, and workflow history into PostHog, so you can analyze your data catalog alongside your product data.

## Prerequisites

You need an AWS account with Glue enabled in the region you want to sync from.

You also need an IAM user or role with the required Glue read permissions. The exact permissions depend on which tables you want to sync – see [Creating IAM credentials](#creating-iam-credentials) below.

Lake Formation permissions can also restrict catalog access. If you use Lake Formation, make sure the IAM principal has read access to the databases and tables you want to sync.

## Adding a data source

1.  In PostHog, go to the [Sources tab](https://app.posthog.com/data-management/sources) of the data pipeline section.
2.  Click **\+ New source** and click **Link** next to this source.
3.  Enter your credentials (see [Configuration](#configuration) below) and click **Next**.
4.  Select the tables you want to sync, choose a sync method and frequency, then click **Import**.

Once the syncs are complete, you can start querying this data in PostHog.

When linking AWS Glue Data Catalog, you'll need:

-   **AWS access key ID** – the access key ID for your IAM user or role (starts with `AKIA...`).
-   **AWS secret access key** – the corresponding secret access key.
-   **AWS session token** (optional) – only required if you are using temporary STS credentials. Temporary credentials expire, so replace them before the next scheduled sync.
-   **AWS region** – the region to sync from (for example, `us-east-1`). Each source connection syncs from one region.

## Creating IAM credentials

1.  In the AWS Console, go to **IAM** > **Users** and create a new user (or use an existing one).
2.  Attach a policy with the permissions for the tables you want to sync:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabases",
        "glue:GetTables",
        "glue:GetPartitions",
        "glue:GetTableVersions",
        "glue:GetJobs",
        "glue:GetJobRuns",
        "glue:GetCrawlers",
        "glue:ListCrawls",
        "glue:ListWorkflows",
        "glue:GetWorkflowRuns",
        "glue:ListDataQualityResults"
      ],
      "Resource": "*"
    }
  ]
}
```

You can scope this down to only the permissions you need:

| Tables | Required permissions |
| --- | --- |
| `databases`, `tables`, `partitions`, `table_versions` | `glue:GetDatabases`, `glue:GetTables`, `glue:GetPartitions`, `glue:GetTableVersions` |
| `jobs`, `job_runs` | `glue:GetJobs`, `glue:GetJobRuns` |
| `crawlers`, `crawls` | `glue:GetCrawlers`, `glue:ListCrawls` |
| `workflow_runs` | `glue:ListWorkflows`, `glue:GetWorkflowRuns` |
| `data_quality_results` | `glue:ListDataQualityResults` |

3.  Create an access key for the user under **Security credentials** > **Access keys**.
4.  Copy the **Access key ID** and **Secret access key** – you'll need both when linking the source in PostHog.

## Sync modes

Each table can be synced in one of several modes, depending on what the source supports:

-   **Webhook** (when available) – the source pushes changes to PostHog in real time. Fastest freshness, lowest ongoing cost, and the only mode that reliably captures updates and deletes.
-   **Incremental** – only new or updated rows are synced on each run, using a cursor field (such as an `updated_at` timestamp). Cheaper than a full refresh, but deletes aren't captured.
-   **Append only** – new rows are appended using a cursor field; existing rows are never updated. Ideal for immutable, append-only tables like event logs.
-   **Full refresh** – the whole table is reloaded on every sync. Use it when a table has no reliable cursor or when you need deletions reflected.

See [sync methods](https://posthog.com/docs/data-pipelines/sources.md#sync-methods) for a full explanation of how each mode works and how to choose between them.

All AWS Glue Data Catalog tables are full refresh. Each sync replaces the contents of the table.

## Configuration

| Option | Description |
| --- | --- |
| `AWS access key ID` **Type:** text **Required:** True |  |
| `AWS secret access key` **Type:** password **Required:** True |  |
| `AWS session token` **Type:** password **Required:** False | Temporary credentials expire. Replace them before the next scheduled sync. |
| `AWS region` **Type:** text **Required:** True |  |

## Supported tables

| Table | Description | Sync method | Incremental field | Primary key |
| --- | --- | --- | --- | --- |
| `databases` | Databases in the account's default Data Catalog. | Full refresh | — | — |
| `tables` | Table definitions, columns, and storage metadata for each database. | Full refresh | — | — |
| `partitions` | Partition values and storage metadata for each table. | Full refresh | — | — |
| `jobs` | Job definitions and execution settings. | Full refresh | — | — |
| `job_runs` | Job execution history, status, duration, and resource usage. | Full refresh | — | — |
| `crawlers` | Crawler settings, state, and the latest crawl result. | Full refresh | — | — |
| `crawls` | Crawl history for each crawler, including duration, status, and DPU usage. | Full refresh | — | — |
| `table_versions` | Schema and metadata history for each catalog table. | Full refresh | — | — |
| `workflow_runs` | Workflow run history, status, timing, and action statistics. | Full refresh | — | — |
| `data_quality_results` | Data quality evaluation runs, with the evaluated table and the job run. | Full refresh | — | — |
| `data_quality_results_v2` | Data quality results published from any supported engine, with the evaluated table and run context. | Full refresh | — | — |
| `column_statistics` | Column statistics for each catalog table, such as null counts, distinct values, and value ranges. | Full refresh | — | — |
| `table_optimizer_runs` | Compaction, snapshot retention, and orphan file deletion runs for each Iceberg table. | Full refresh | — | — |
| `sessions` | Interactive sessions, with status, duration, and compute use. | Full refresh | — | — |

## Troubleshooting

-   If the connection fails with an access denied error, check that the IAM user or role has the required Glue read permissions. If you use Lake Formation, also check that the IAM principal has access to the catalog resources.
-   If you see a credential error, check that the access key ID and secret access key are correct and that the key is still active in IAM.
-   If you see a signature error, re-enter the secret access key. If you are using temporary credentials, the session token may have expired.
-   If the session token has expired, create new temporary credentials and reconnect the source.
-   If you see an error about Glue not being enabled, enable AWS Glue for the account and region, then reconnect.
-   If a table syncs no rows, the credential may not have permission to read that data. Check the IAM policy and any Lake Formation restrictions.

If your sync is failing or data looks wrong, see the [Data warehouse troubleshooting guide](https://posthog.com/docs/data-warehouse/troubleshooting.md). If that doesn't help, [contact support](https://us.posthog.com/#panel=support%3Asupport%3Adata_warehouse%3A%3Atrue) – we're happy to help.