Data Export Fulfillment Processor

The Data Export Fulfillment Processor is a nightly batch workflow (introduced in v1.0.33) that automatically processes pending data export requests, generates downloadable archives, and manages their full expiry lifecycle — with no manual intervention required.

How It Works

The workflow runs once per day at 01:00 UTC on the schedule 0 1 * * *. Each run performs two passes:

Pass 1 — Fulfill Pending Requests

1. Queries data_export_requests for all records where status = 'pending'.
2. For each request, compiles a complete data archive containing the user's:
   • Account information (users)
   • Projects (projects)
   • Features (features)
   • Pipeline runs (pipeline_runs)
   • Releases (releases)
3. Produces the archive in both JSON and CSV formats.
4. Uploads the archive to S3-compatible object storage.
5. Updates the request record with:
   • downloadUrl — a signed URL pointing to the archive
   • expiresAt — set to 7 days from the time of processing
   • status → ready

Pass 2 — Expire Stale Downloads

1. Queries data_export_requests for all records where status = 'ready' and expiresAt < NOW().
2. Updates each matched record: status → failed.

This ensures users are never left with a silently broken download link — the failed status signals clearly that the archive has expired and a new export request is needed.

Request Status Lifecycle

┌─────────┐     nightly batch     ┌───────┐
│ pending │ ──────────────────►  │ ready │
└─────────┘                       └───┬───┘
                                      │  expiresAt < NOW()
                                      ▼
                                  ┌────────┐
                                  │ failed │
                                  └────────┘

Status	Meaning
pending	Export requested; not yet processed
ready	Archive generated; downloadUrl is valid
failed	Download link has expired; re-request required

Archive Contents

Each export archive includes a snapshot of all data associated with the requesting user across the following entities:

Entity	Description
users	User account and profile data
projects	All projects owned by or associated with the user
features	Feature records linked to those projects
pipeline_runs	CI/automation pipeline execution history
releases	Release records and metadata

Archives are available in both JSON (structured, machine-readable) and CSV (flat, spreadsheet-compatible) formats.

Download Expiry

Download links are valid for 7 days from the time the export is fulfilled. After that window, the request transitions to failed and the archive is no longer accessible. Users should download their archive promptly after receiving confirmation that it is ready.

Schedule

Property	Value
Trigger	Scheduled (cron)
Cron expression	0 1 * * *
Runs at	01:00 UTC, every day
Workflow type	nightly_batch