Batch Rescreen Engine

The batch rescreen engine runs nightly (and on demand) to re-screen all monitored individuals against the current OFSI consolidated sanctions list. It is the core mechanism that ensures your compliance posture stays current as the sanctions list changes.

How it works

1. The current active sanctions list is loaded once from the database into memory.
2. All non-deleted monitored people are processed in chunks of 100, using a cursor on people.id.
3. Each person is screened against the list using the tenant's configured match threshold and cross-validation settings (DoB, nationality).
4. Match records are updated: pending matches are refreshed; reviewed (confirmed/dismissed) matches are preserved.
5. Person statuses (matched, clear) are updated accordingly.
6. A rescreenRuns record is created at the start and updated after every chunk and at completion.

Chunked cursor iteration (v0.1.38+)

As of v0.1.38, the engine uses cursor-based pagination instead of a single full-table scan. People are fetched in batches of 100 rows using id > lastCursorId ORDER BY id ASC.

Why cursor-based?

Approach	Problem at scale
Full-table scan	Memory exhaustion in serverless; Neon Postgres query timeouts
Offset pagination	Rows skipped or double-processed when inserts occur mid-run
Cursor (primary key)	Stable, index-backed, insertion-order-safe

The primary key (id) is monotonically increasing and fully indexed, making it an ideal cursor — each chunk boundary is deterministic regardless of concurrent writes.

Loop termination

The chunk loop exits when:

• A chunk returns fewer than 100 rows — this is the final page.
• The first chunk is empty — no people to process.

This avoids an extra database round-trip to confirm the end of the dataset.

Per-chunk checkpointing

After processing each chunk, the engine writes the following fields to the rescreenRuns record:

Field	Description
lastCursorId	The highest people.id processed so far
chunksProcessed	Running count of completed chunks
elapsedMs	Milliseconds since the run started

This means:

• Real-time monitoring: Operators can observe run progress from the dashboard without waiting for completion.
• Failure recovery: If a run fails mid-way, the last persisted cursor can be used to resume or re-run from the point of failure.

BatchRescreenResult

The function returns a BatchRescreenResult object on completion:

export interface BatchRescreenResult {
  runId: number;             // ID of the rescreenRuns record
  totalPeopleScreened: number;
  newMatchesFound: number;
  matchesCleared: number;
  errorCount: number;
  chunksProcessed: number;   // Added in v0.1.38
  durationMs: number;
}

chunksProcessed is also recorded in the completion audit log entry alongside chunkSize (100), giving a full picture of run mechanics for audit purposes.

Triggering a batch rescreen

A batch rescreen is triggered automatically:

• Every night as part of the nightly OFSI list sync.
• When a new list version is ingested.

It can also be triggered manually. The triggeredBy field on the rescreenRuns record records the source (nightly_sync, list_update, or manual).

Per-tenant screening settings

Each user's monitored people are screened using that user's own settings:

• Match threshold: Minimum fuzzy-match confidence score to flag a match.
• DoB cross-validation: Whether to use date of birth to confirm or discount matches.
• Nationality cross-validation: Whether to use nationality to confirm or discount matches.

Settings are cached in memory for the duration of the run to avoid redundant database lookups when many people belong to the same tenant.

Match preservation

The engine distinguishes between pending and reviewed matches:

• Pending matches are deleted and re-created fresh on each run.
• Reviewed matches (confirmed or dismissed) are never deleted — reviewer decisions are always preserved.

A person is only moved to clear status if they have no remaining matches at all (including reviewed ones).