Stuck Archive Watchdog

Overview

The Stuck Archive Watchdog is a scheduled maintenance job that automatically detects tenancy archives that have stalled in a pending state and escalates them to the organisation admin. It prevents silent failures from blocking tenancy close-outs and ensures that every stalled archive is surfaced and actioned promptly.

---

How it works

Schedule

The watchdog runs once per day at 04:00 UTC via a cron schedule (0 4 * * *). No manual trigger is required.

Detection logic

On each run, the job queries tenancy_archives for records that meet both of the following conditions:

Condition	Value
status	'pending'
createdAt	more than 1 hour ago

Any record matching both conditions is considered stuck.

Remediation steps

For every stuck archive detected, the watchdog performs the following actions in sequence:

1. Updates the archive record

   • Sets status → 'error'
   • Writes a human-readable errorMessage to the record explaining the failure reason.

2. Fires an urgent in-app notification to the organisation admin

   • Includes the archive ID and the tenancy address.

3. Fires an urgent email notification to the organisation admin

   • Includes the archive ID and the tenancy address.

---

Affected entities

• tenancyArchives — records are read and updated (status + errorMessage).
• orgMembers — the organisation admin recipient is resolved from this entity.
• notifications — in-app notification records are created here.

---

What admins should do

When you receive a stuck-archive notification (in-app or email):

1. Note the archive ID and tenancy address included in the notification.
2. Navigate to the archive record in the admin panel.
3. Investigate the root cause — common causes include upstream data issues, third-party API timeouts, or missing documents.
4. Re-trigger the archive process once the underlying issue is resolved, or contact support if the cause is unclear.

Note: Because the watchdog only runs once per day, an archive can be stuck for up to ~25 hours before it is flagged (if it stalls just after a watchdog run). If you suspect an archive is stuck sooner, you can check its status field directly in the admin panel.

---

Notifications reference

In-app notification

• Priority: Urgent
• Recipient: Organisation admin
• Content: Archive ID + tenancy address

Email notification

• Priority: Urgent
• Recipient: Organisation admin
• Content: Archive ID + tenancy address

---

Version introduced

This feature was introduced in v0.1.227.