Export Scheduling & Timing Guide

Audience: CS, SE, and Engineering teams
Last verified against codebase: Feb 2026

Widget Exports

Widget exports are scheduled via a Kubernetes CronJob that runs every hour (0 * * * *). This triggers the Airflow widget-exporter DAG, which runs the export:widget[company_id] rake task as a K8s pod. Within each run, CustomScheduler.time_to_run? determines which configs are due.

How Scheduling Works Per Config

Each WidgetExportConfig has two key fields in its metadata JSON:

CustomScheduler evaluates whether a config is due based on:

1. The config's export_hour (in company timezone, NOT UTC)

2. The config's interval (default 24h, but configurable down to 1h)

3. The config's last_job_started_at timestamp

Important: There are no fixed "3 AM / 11 AM UTC" slots. Those were legacy values from before the migration to per-config scheduling. Each config runs at its own export_hour in the company's timezone.

Historical Context

The old system used an export_time field with values "morning" (3 AM UTC) and "evening" (11 AM UTC). A migration script (scripts/widget_export_configs_migration.rb) converted these to timezone-relative export_hour values. The old export_time field still exists in the schema with a TODO to remove it.

How to Check a Customer's Widget Export Schedule

1. Look up the WidgetExportConfig in Retool (Admin > Widget Export Configs)

2. Check the metadata JSON for export_hour and interval

3. Look up the company's timezone to convert export_hour to UTC

4. If export_hour is missing, it defaults to 0 (midnight in company timezone)

5. If interval is missing, it defaults to 24 (once daily)

Manual Runs

Manual / one-off widget exports use a separate path from the scheduled pipeline:

• Triggered via Retool, which calls the schedule_export controller action

• This enqueues a WidgetExportJob on the Sidekiq widget_exporter queue

• The job creates a temporary (non-persisted) WidgetExportConfig from temp_config_params

• A dedicated Sidekiq deployment (sidekiq-widget-exporter-deployment.yaml) processes this queue

Key distinction: Scheduled exports run via Airflow/K8s pods (rake task). Manual exports run via Sidekiq. They are different execution paths.

Data Exports

Data exports are currently scheduled via cron jobs defined in cron_job/jobs/production/data_exporter_jobs.rb.

Standard Schedule (Non-DH Customers)

DH-Specific Schedule

DH (Dentsu Happiness) has custom timing due to their data pipeline dependencies:

Why DH is different: After the Mozek migration, DH exports needed to align with their internal data refresh cadence. This was set via tenant-level override (see Shortcut sc-53018).

Resource Allocation

Data export pods have different memory allocations based on channel size:

Primary vs Secondary Exports

• Primary: Exports the full configured duration (e.g., last_30d, last_90d). Runs once daily.

• Secondary: Exports only the last 3 days. Useful for downstream systems that need fresher data without waiting for the full primary run.

Upcoming: Mozek Migration for Data Exports (sc-52754)

Status: In development (branch sc-52754/move-data-exporter-crons-to-mozek, not yet merged). Author: Chinmay Relkar.

What's Changing

Data export scheduling is being moved from adwyze cron jobs into Mozek's DAG-based orchestration framework. This adds data exports as a first-class workload type in Mozek, alongside ad channel data pipelines.

New Mozek Tenant Types

Two new tenant types have been added to Mozek.V3.Orchestrator.Tenant:

New DAG Definitions

Two single-node DAGs have been created:

• data_export — triggers primary export via Sidekiq executor

• data_export_secondary — triggers secondary export via Sidekiq executor

Both DAGs use dynamic split args to resolve tenant_external_id from the tenant's external ID, and both set tenant_source: "clarisights".

Failure Recovery

Both DAGs are configured with:

• Retry window: 5 days (432,000 seconds)

• Base retry interval: 10 minutes (600 seconds)

This is a major improvement over the current cron-based approach, which has no built-in retry or failure recovery.

What This Means for the Team

Once merged, this migration brings several benefits:

• Automatic failure recovery — failed exports will be retried within the 5-day window, instead of waiting for the next day's cron

• Non-overlapping work guarantee — Mozek prevents duplicate export runs

• Granular checkpointing — lower cost of failure vs fire-and-forget crons

• Visibility & alerts — export status visible in Mozek dashboard

• Automatic data gap fixing — Mozek detects and fills missed export windows

What's NOT Changing (Yet)

• The actual export logic in adwyze (DataExportJob, CsvTarget, S3/GCS uploads) stays the same

• Widget exports are NOT part of this migration (they stay on Airflow)

• Export configs are still managed in Retool via DataExportConfig

• The Mozek DAGs use the Sidekiq executor, so they ultimately enqueue the same export jobs

Related Work

This is part of a broader pattern (documented in ADR 0045) of migrating workloads from Airflow/cron to Mozek. The Adjust analytics channel is also being migrated for similar reasons (better failure handling, reduced redundant work).

Common Questions from Support Threads

"When will the customer see their widget export?"

Check the customer's WidgetExportConfig for export_hour and interval, then convert to UTC using the company's timezone. The pipeline runs every hour and picks up configs that are due. Typically the export completes within 1-2 hours of the scheduled time, depending on queue depth.

"When will the customer see their data export?"

• Data export primary: After 2 PM UTC (or 9:30 AM UTC for DH)

• Data export secondary: After midnight UTC (or 4:30 PM UTC for DH)

"Can we change the export time for a customer?"

• Widget exports: Yes — update export_hour in the WidgetExportConfig metadata via Retool. Remember this is in the company's timezone, not UTC.

• Data exports: Requires a tenant-level cron override or a custom cronjob entry. Coordinate with engineering.

"The export didn't run today"

Check in this order:

1. Widget exports:

   • Airflow UI — was the widget-exporter DAG triggered?

   • Check CustomScheduler logic — is the config's export_hour + interval due?

   • Check last_job_started_at on the config

2. Data exports:

   • DataExportLog table — check for STARTED/COMPLETED/FAILED status

   • Sidekiq dashboard — is the job stuck in retry?

   • Mozek dashboard (once migration is complete)

Architecture Summary

Widget Export Pipeline (Current)

K8s CronJob (every hour, 0 * * * *)
  → Airflow queue (ingestion:queue rake task)
    → Airflow DAG Runner (widget-exporter DAG)
      → K8s Pod runs export:widget[company_id] rake task
        → CustomScheduler.time_to_run? per config
          → WidgetExporter.export (inline, NOT Sidekiq)

Widget Export Manual Run

Retool → schedule_export controller
  → WidgetExportJob.perform_async (Sidekiq)
    → widget_exporter queue
      → WidgetExporter.export with temp config

Data Export Pipeline (Current)

K8s CronJob (cron schedule per channel)
  → data_exporter:for_channel[channel] rake task
    → DataExportJob.run
      → Per-config export via CsvTarget → S3/GCS

Data Export Pipeline (After Mozek Migration)

Mozek Orchestrator
  → data_export / data_export_secondary DAG
    → Sidekiq executor
      → DataExportJob.run (same export logic)
        → Per-config export via CsvTarget → S3/GCS
  + Automatic retry (5-day window, 10-min intervals)
  + Non-overlapping guarantees
  + Gap detection & fixing

Key Code References

• Widget scheduling logic: lib/widget_exporter.rb → export_configs_to_run

• CustomScheduler: lib/custom_scheduler.rb

• Widget Airflow DAG: python/ingestion_pipeline/airflow/dags/dags/generated/widget_exporter/widget_exporter.py

• Widget manual export: app/sidekiq/widget_export_job.rb + app/controllers/widget_export_configs_controller.rb

• Data export crons: cron_job/jobs/production/data_exporter_jobs.rb

• Mozek export DAGs (branch): test/mozek/v3/fixtures/dags/data_export.json

• Mozek tenant types: lib/mozek/v3/orchestrator/tenant.ex

• ADR 0045 (scheduling migration rationale): adrs/0045-analytics-scheduling.md