Skip to content
Mark Warren
Menu

All work

Regression watchdog

A regression watch that speaks only when something got worse

A weekly re-run of the AI-visibility audit for each enrolled client site, with a second run required to confirm a drop before anyone is told.

How it fits together

  1. Inputs

    • Enrolled sites

      A site is watched only after an admin presses Watch on the dashboard. Nothing is enrolled by default.

    • Two daily schedules

      One starts the due audits, and a later one asks for the digest. The scheduler holds no rules of its own.

      • n8n

  2. Engine

    Compare with the last run

    Each new audit is compared with the previous comparable run of the same site, by rules written in TypeScript with tests.

    • TypeScript
    1. Confirm before alerting

      A drop opens a candidate. The next run confirms it or drops it as a flap, and a candidate alone is never sent.

    2. Fair comparisons only

      A new scoring version or a different measured page resets the baseline, and a check that could not run is never counted as a regression.

    3. Shares the refresh slot

      The watchdog and the dashboard's Refresh button take the same per-site slot, so the two never start overlapping audits of the same site.

  3. Outputs

    • Operator digest

      Confirmed regressions, recoveries, and sites that could not be measured, with a triage link for each client.

      • Telegram

    • Triage board

      FirstCoast sees every event. A client member sees one only after it has been triaged, enforced by the database.

      • PostgreSQL

Guardrails

  • No model calls and no paid APIs
  • Rows are marked sent only after Telegram accepts the message
  • Hard limits on sites, audits per cycle, and digest length, in code

The problem

A site that passed the AI crawler check last month can fail it today because someone edited a robots file, a CDN rule changed, or a CMS update dropped the structured data. The audit engine finds that, but only when someone runs it. An alert that fires on every wobble gets ignored, so the watch has to be quiet until a drop is real.

What I built

  • A weekly schedule that re-runs the existing deterministic audit for each site an admin has enrolled, oldest data first, with pending confirmations ahead of everything else.
  • Diff rules that open a candidate on a status drop or a large score drop, confirm it on the next run, and close it when the check recovers.
  • A could-not-run path: a failed trigger is reported at once, and an audit that never came back is reported at the next digest instead of being read as a pass.
  • A Telegram digest for the operator, where each row is marked sent only after Telegram accepts the message, so a failed send rides the next digest instead of being lost.
  • A triage board in the client dashboard, where history cannot be deleted and clients see only what FirstCoast has reviewed.

How I built it

The rules, the cycle phases, and the SQL live in the Next.js dashboard I built for Mulberry Marketing Communications, behind one internal endpoint. The n8n workflow only keeps time and relays the digest. Tests cover the rules and phases, and a separate suite runs against a real PostgreSQL database to check row-level security, triage visibility, and that two schedulers cannot claim the same slot. It went live in November 2026 after dry runs, and the daily morning health check flags it if either schedule stops running.

What it shows

I design monitoring around the person reading the alert: quiet by default, confirmed before it speaks, and honest when it could not measure.