Getting Started with Export Pipelines

Export Pipelines let you schedule automated, recurring data exports from Clarisights to your own cloud storage (Amazon S3 or Google Cloud Storage). You define what data to export, where to send it, and when — and Clarisights handles the rest.

Key Concepts

An export pipeline is composed of four building blocks:

Each time a schedule fires, it creates an Export Run — a record of the execution with status, file URLs, and byte counts.

What's Supported

• Destinations: Amazon S3, Google Cloud Storage

• Format: CSV

• Compression: gzip (default) or none

• Scheduling: Any cron expression with a minimum interval of 1 hour

• Date ranges: Presets like today, yesterday, last_7d, last_30d, or explicit start/end dates

Quick Walkthrough

Here's the typical flow to set up your first export pipeline:

1. Create a Destination — Configure your S3 or GCS bucket with the appropriate credentials and permissions. See Configuring Export Pipeline Destinations & Permissions.

2. Create a Pipeline — Give it a name and select which data sources (channels) to include (e.g., Google Ads, Facebook, TikTok).

3. Create a Manifest — Define one or more extracts, each specifying the metrics, dimensions, and optional filters you want. The available metrics and dimensions are shown dynamically based on your selected data sources.

4. Create a Schedule — Set a cron expression (e.g., 0 12 * * * for daily at noon), choose a date range preset, link it to your destination, and activate it.

5. Monitor Runs — Check the export runs list to see status (running, succeeded, failed), file sizes, and download links.

Pipeline Lifecycle

• Active: Schedules fire normally and create export runs.

• Paused: No exports are triggered. Resume at any time by setting status back to active.

• Deleted: Soft-deleted. The pipeline and its data are retained for 30 days before permanent removal.

You can pause and resume both individual schedules and entire pipelines independently.

Manifest Versioning

When you create a new manifest for a pipeline, the previous manifest enters a 30-day grace period. During this time, both the old and new manifests are usable. After 30 days, the old manifest expires and is eventually cleaned up.

This means you can update your export shape (add/remove metrics, change filters) without breaking downstream consumers that still expect the old format.

What Gets Exported

Each export run produces:

• A CSV data file (optionally gzip-compressed) per extract, chunked into 100,000-row files if needed

• A metadata file (.meta.json) alongside each data file containing column mappings, date range, and compression info

Files are organized in your bucket based on your set filename template as:
{path_prefix}/from_clarisights/{company_name}/{channel_name}/{extract_id}/{timestamp}_{start_date}_{end_date}.csv.gz

How is this different from Legacy Data Exporters / Widget Exports?

Note: Data Exporters and Widget Exports will be sunset in August 2026. We recommend migrating to Export Pipelines before this date.

For detailed setup instructions, see:

• Configuring Export Pipeline Destinations & Permissions

• Managing Export Schedules & Manifests