AI Assessment Feedback-to-Improvement Loop

Available from: v0.1.350

Overview

The AI Assessment Feedback-to-Improvement Loop is an event-driven workflow that automatically captures the outcome of every adjudicated dispute and uses it to build a running picture of AI assessment accuracy — on a per-organisation basis.

Once a dispute is resolved through the post-adjudication process, the outcome's key signals (reasoningTags and awardRatio) are extracted and written into a performance snapshot. Over time, these snapshots reveal how well AI deduction assessments align with real adjudication decisions.

---

How It Works

Trigger

The workflow is activated by the ai.feedback.dispute_outcome event, which is already emitted automatically when dispute-outcome-post-adjudication completes. No configuration or manual triggering is needed.

Event: ai.feedback.dispute_outcome
Type:  Sync

Execution Flow

dispute-outcome-post-adjudication
        │
        └─► fires: ai.feedback.dispute_outcome
                        │
                        ▼
            Read disputeOutcomes
              - reasoningTags
              - awardRatio
                        │
                        ▼
            Update ai_performance_snapshot (per org)
              or fallback: analyticsSnapshots
                        │
                        ▼
            Persist updated metrics:
              - avg award ratio
              - tag frequency distribution
              - win/loss patterns by deduction category

Metrics Tracked

Metric	Description
Average Award Ratio	Running average of how much of the claimed deduction was awarded, across all disputes for the organisation.
Tag Frequency Distribution	How often each reasoningTag appears in adjudicated outcomes — surfaces which reasoning patterns correlate with success or failure.
Win/Loss Patterns by Deduction Category	Dispute outcomes grouped by the type of deduction (e.g. cleaning, damage, rent arrears), showing where AI assessments are most and least aligned with adjudicator decisions.

---

Entities

Entity	Role
disputeOutcomes	Source of truth for adjudication results — provides reasoningTags and awardRatio.
aiDeductionAssessments	The original AI assessments being evaluated for accuracy.
analyticsSnapshots	Fallback store for performance snapshots when ai_performance_snapshot is unavailable.
usageEvents	Used to log workflow execution for auditing and billing purposes.

---

Storage: ai_performance_snapshot

Each organisation has its own ai_performance_snapshot record that is updated (not replaced) on every event. This means metrics accumulate incrementally — each new dispute outcome shifts the running averages rather than overwriting previous data.

If no snapshot exists for the organisation yet, one is created automatically on first event receipt.

---

Relationship to Other Features

• Dispute Outcome Post-Adjudication — The upstream process that emits the trigger event. This feature requires it to be active.
• AI Deduction Assessments — The assessments being measured. Accuracy metrics produced here will inform future model tuning and threshold calibration.
• Analytics Snapshots — Used as the fallback persistence layer; also queryable for reporting dashboards.

---

Notes

• This workflow runs synchronously within the post-adjudication event pipeline.
• No additional setup is required — the trigger event is already emitted by the existing adjudication flow.
• Snapshots are scoped per organisation, so metrics do not bleed across tenancies or landlord accounts.