Skip to main content

​
Overview

Data Warehouse Exports write your raw Cometly data as Parquet files to an object storage bucket you own — Amazon S3 or Google Cloud Storage — or load it directly into a warehouse table — Snowflake or BigQuery. Point your warehouse or lake tooling — Snowflake, BigQuery, Databricks, Athena, DuckDB, and others — at the bucket to load the data on your own schedule, or let Cometly load it for you. You keep the raw, unaggregated data and model it however you like on your side.
This is different from the API exports in the API Reference (Create Event Export, Create Contact Export, Create Company Export). Those return gzipped NDJSON for ad-hoc download through the API. Data Warehouse Exports deliver columnar Parquet files to your own object storage for bulk, recurring loads. See Exports vs Pagination for the API-side options.

​
How it works

1

Add a destination

Either a bucket Cometly writes Parquet files to — S3 (Set up an S3 destination) or GCS (Set up a Google Cloud Storage destination) — or a warehouse Cometly loads into directly — Snowflake (Set up a Snowflake destination) or BigQuery (Set up a BigQuery destination).
2

Create an export

Choose a destination, a dataset, a cadence, and a window. See Create an export.
3

Get the data

For S3 or GCS, Cometly writes Parquet files under your chosen path and you point your warehouse or lake at the bucket. For Snowflake or BigQuery, Cometly loads each run into a table in your database.

​
Datasets

Each export unloads exactly one dataset. To export several datasets, create several exports — each with its own cadence, window, destination, and columns.

​
Cadence and windows

An export runs on a cadence and covers a window of data. The two are independent: cadence is how often a run fires; the window is how much each run pulls. A trailing window is anchored to “now” at run time — “last 7 days” means the 7 days ending at the moment the run fires, not a fixed calendar range. Every recurring run re-exports the full window (not just what changed), so runs are self-healing and you deduplicate on a stable key on your side.
A run only re-exports rows that are still inside the window. To refresh data older than the window (for example, after a backfill or a late restatement), create a one-time export over the date range you want. The common pattern is a one-time backfill plus a recurring incremental export.

​
Column conventions

These conventions hold across every dataset, so the same column name means the same thing in every export.
  • Timestamps are UTC. Every *_utc column (and created_at / updated_at) is a UTC timestamp. The one exception is the ad-analytics metric date, which is a local calendar date — see that dataset’s page.
  • Locked vs optional columns. Some columns are locked (always exported — the ones that keep the data coherent, such as row identity and join keys); the rest are optional and you pick them at creation. Each dataset page marks which is which.
  • Cometly ID vs platform ID. Ad entities (account, campaign, ad set/ad group, ad) are carried as IDs only — never names. Each level exposes two columns:
    • cometly_<level>_id — Cometly’s own internal ID for the entity. Stable, and the reliable key for joining one export to another (e.g. touchpoints to ad analytics).
    • platform_<level>_id — the advertising platform’s own ID for the same entity (the Facebook/Google/etc. ID). The key you join to your platform-side data; may be null.
    Resolve campaign/ad names on your side from these IDs.
  • JSON-array columns. A few columns (the contacts identity-graph columns) are exported as JSON-encoded arrays. They’re optional and default to off.

​
Where to manage exports

In the Cometly dashboard, each destination type is its own integration under the Warehouse category on the Integrations page: go to Integrations → Amazon S3, Integrations → Google Cloud Storage, Integrations → Snowflake, or Integrations → BigQuery, depending on the destination type. Each page has a Destinations section scoped to that type, and an Exports table that lists every export across all destination types — so you can manage all your exports from any of the four pages. This feature is available to space administrators.