Skip to main content

​
Overview

A GCS destination is a reusable connection to a Google Cloud Storage bucket you own. Like an S3 destination, Cometly writes Parquet files to your bucket and stops — there’s no load phase. You load the files into BigQuery, Databricks, or another engine yourself, on your own schedule. Cometly authenticates with an HMAC interoperability key, not a service-account JSON key. You generate one for a service account that has write access to your bucket.

​
Prerequisites

  • A Google Cloud project with permission to create Cloud Storage buckets, service accounts, and IAM bindings.
  • A space administrator role in Cometly.

​
Create the bucket and credentials

1

Create or choose a bucket

In the Google Cloud Console, create a Cloud Storage bucket (or choose an existing one) for Cometly exports, and note its name.
2

Create a service account

Create a dedicated service account (for example, cometly-exports) for programmatic access only.
3

Grant it access to the bucket

The export only ever lists the bucket’s prefix and writes objects — it never deletes anything. Create a custom role with exactly these permissions:
  • storage.objects.list — the export lists its output prefix
  • storage.objects.create — the export writes the Parquet files
  • storage.multipartUploads.create, storage.multipartUploads.abort, and storage.multipartUploads.listParts — large files upload in parts through the S3-compatible API
Grant the role to the service account on the bucket (Bucket → Permissions → Grant access), not on the project.Because the role excludes storage.objects.delete, the connection check on save can’t clean up its tiny .cometly-connection-test/… probe object — that’s expected; remove it manually if you like.
4

Generate an HMAC interoperability key

In Cloud Storage, go to Settings → Interoperability, and create an HMAC key for the service account. Save the Access key and Secret — the secret is shown only once.
5

Add the destination in Cometly

In the dashboard, go to Integrations → Google Cloud Storage → Add destination and enter:
​
string
required
A label for this destination, for example “Production warehouse”.
​
string
required
The GCS bucket name from Step 1.
​
string
required
The access key from the interoperability key you generated, for example GOOG1E….
​
string
required
The secret from the interoperability key. Shown only once when you generate it in Cloud Storage.
Cometly runs a quick, best-effort connection check on save. Because permissions scoped to a prefix can legitimately deny a whole-bucket check, a failed check is logged but never blocks the create — if your key and bucket are correct, the destination still works even if the check couldn’t confirm it.
Your HMAC secret is encrypted at rest and never shown again. When editing a destination, leave the secret blank to keep the existing key.
Scope the service account to only the bucket used for exports. Avoid reusing a broadly privileged service account.

​
File layout

A GCS destination writes the same Hive-style partitioned Parquet layout as an S3 destination — see File layout (S3 and GCS destinations) for the path structure, partition columns, and the _SUCCESS marker.

​
Using with BigQuery

BigQuery can read these files directly from your bucket, without a load step:
  • Create an external table over the …/<dataset>/space_id=<id>/ prefix to query the Parquet files in place.
  • Or run a load job to copy the data into a native BigQuery table — Cloud Storage load jobs are free.
Prefer a native load without managing external tables or load jobs yourself? Use a BigQuery destination instead — Cometly loads directly into a table in your dataset, the same way the Snowflake destination does, with no file layout to manage.

​
Next step

Create an export

Choose this destination, a dataset, and a date range.