Sentinel

1. Context

The Sponic monorepo has grown a wide surface — two deployed apps (apps/garden, apps/control), Supabase Postgres with 50+ migrations and ~15 critical tables, three pg_cron jobs, two Cloudflare Workers, ~15 external services, a headless claude -p runner on Oracle Phoenix — with near-zero automated verification. CI runs only tsc + eslint + build + image-gen lint. There are no unit or e2e tests, no health endpoints, no uptime monitoring, no Sentry/Datadog, no dependency scanning, no pre-commit hooks. The only smoke test is a manual smoke-image-gen.mjs.

The tasks system already has a queue + headless claude -p runner + cost tracking + activity log on Oracle Phoenix. Sentinel reuses concepts (queueing, persisted runs, cost telemetry) but runs on a separate orchestrator on alpu.ca so a Sentinel failure can't break the tasks UI and vice versa.

Goal

Ship a system that:

1. Runs automated checks across every surface daily.
2. Auto-extends coverage as new dev work lands (without manual setup per feature).
3. Presents activity, agents, findings, coverage in a dedicated intranet section called Sentinel, with first-class transparency into every multi-step run (prompts, models, handoff artifacts, costs).
4. Generates structured recommendations without auto-acting on them. (Test-coverage findings can be approved & queued to the prompt-runner via a manual gate.)

2. Architecture — four layers

1. Probes — deterministic checks. Cheap, scriptable, structured-output. Live in apps/control/src/lib/sentinel/probes/. Each probe is a TS function (ctx) => ProbeResult. Probes can attach artifacts (HTTP response bodies, screenshots, stack traces, log excerpts) which are stored either inline or in R2. Each probe is also exposed to the Agent SDK as a tool — so probes serve double duty: directly callable in deterministic agents, and tool-callable by LLM-driven agents.
2. Reasoner — LLM execution layer with two modes:
   • One-shot (reasoner/oneshot.ts): direct Anthropic API call with structured output schema. Used by deterministic agents that just need a brief summary + verdict over already-collected probe results. Cheap, fast, no tool calling.
   • Agent-loop (reasoner/agent-loop.ts): Claude Agent SDK initialized with a custom tool catalog (the probes from layer 1, plus a few orchestrator-supplied tools like record_finding, query_supabase, read_file). Used by agents that need to navigate the codebase or chain probes based on what they find — Test coverage and Security passive primarily.
   • Per-agent config selects which mode + model. The mode is stored on the agent record and locked in at run time.
3. Orchestrator — systemd service on alpu.ca. Triggers from cron + an HMAC-signed webhook for manual UI runs. Reads the agent registry from Supabase, dispatches each run to the configured execution mode, collects step events emitted by the SDK (or by the one-shot path), persists every step (prompts, context, output, cost, tool calls, model rationale) to monitoring_run_steps. Source: infra/alpu/sentinel-orchestrator/.
4. UI — new intranet section Sentinel in apps/control. Seven tabs: Overview, Coverage, Agents, Activity, Findings, Runs, Docs. Detailed in §9.

Layers are decoupled: probes are TS functions usable from either execution mode; the orchestrator can be moved (e.g. to Oracle Phoenix later) without touching probes/UI; the SDK version can be upgraded without touching probes or UI.

3. Multi-step transparency

Every agent run is decomposed into discrete steps that are individually persisted. A step is anything with a clear input → output and an attributable model/cost. Examples:

• Probe execution (step_kind = probe) — input = probe config; output = ProbeResult + artifacts. Used by both modes.
• One-shot reasoner call (step_kind = reasoner) — input = collected probe outputs + diff context; output = structured findings. Used only by one-shot agents.
• Agent-loop turn (step_kind = agent_turn) — one round of the SDK loop: model produced a message and optionally invoked tools. Output includes model text + tool-use intents.
• Tool call (step_kind = tool_call) — input = tool name + arguments; output = tool result. Emitted by the SDK whenever the agent calls one of the probes/utilities in its catalog.
• Aggregation (step_kind = aggregate) — combining outputs across steps; rare, used for orchestrator-level deduping.

For agent-loop runs, the SDK emits per-turn and per-tool-call events natively; the orchestrator subscribes to those and writes them straight into monitoring_run_steps. We do not hand-roll the multi-step machinery — we adopt it.

Each step records: kind, model used (for steps where a model was invoked), model rationale (locked in at run time from the agent config), the prompt template id + rendered prompt text or system prompt, the full input context (jsonb), the output (jsonb), tool name + arguments + result for tool calls, references to artifacts produced, references to artifacts consumed (handoff lineage), duration, tokens, cost, status.

In the UI, clicking into any run shows a stepper with all of this. For agent-loop runs, the stepper renders as alternating agent turns and tool calls with arrows showing which turn invoked which tool. For one-shot runs, it's a flat list of probes followed by a single reasoner step. Both surface identically in the UI; the underlying mode is just a chip on the run header.

4. v1 agent set (7 agents)

Each agent owns a domain, has its own model config + execution mode, and registers its probes in monitoring_manifest. Mode = how the orchestrator runs it: one-shot (collect probe results, summarize once) or agent-loop (Claude Agent SDK with tool catalog, iterates).

Plus, outside alpu.ca, a tiny dead-man-switch Cloudflare Worker (apps/control/worker/sentinel-deadman/) that flags a finding row if no orchestrator heartbeat in 25h.

5. Data model — Supabase schema

New migration: apps/control/migrations/20260506_sentinel_schema.sql. All tables admin-only via RLS (role check against app_users.role IN ('oracle','admin')).

monitoring_agents
  id, name, slug, description, surface, owner,
  execution_mode text,            -- 'one-shot' | 'agent-loop'
  model_provider, model_id, model_params jsonb,
  model_rationale text,           -- short "why this mode + model" string
  system_prompt_template_id uuid, -- agent-loop only
  tool_catalog text[],            -- agent-loop only; subset of registered tool names
  max_turns int,                  -- agent-loop only; hard cap on SDK loop iterations
  schedule_cron text, enabled bool,
  daily_cost_ceiling_usd numeric, -- auto-pause if exceeded in 24h window
  timeout_seconds int default 600,
  last_run_id, last_status, last_run_at,
  created_at, updated_at

monitoring_runs
  id, agent_id, trigger ('cron'|'manual'|'deadman'),
  triggered_by uuid (app_users.id, nullable for cron),
  triggered_meta jsonb,
  started_at, completed_at,
  status ('queued'|'running'|'success'|'partial'|'failed'|'cancelled'|'timeout'),
  summary text, cost_usd numeric, total_tokens int,
  models_used text[], step_count int

monitoring_run_steps
  id, run_id, agent_id, step_index int, step_name,
  step_kind ('probe'|'reasoner'|'aggregate'|'tool_call'),
  model_provider text, model_id text, model_rationale text,
  prompt_template_id, prompt_text,
  input_context jsonb, output jsonb,
  artifact_ids uuid[],            -- references monitoring_artifacts
  consumed_artifact_ids uuid[],   -- handoff lineage
  started_at, completed_at, duration_ms int,
  prompt_tokens int, completion_tokens int, cost_usd numeric,
  status ('success'|'fail'|'timeout'|'error'|'skipped'),
  error text

monitoring_prompt_templates
  id, name, version int, agent_slug, step_kind,
  description text, content text, variables text[],
  created_at, deprecated_at,
  unique (name, version)

monitoring_probes
  id, run_id, run_step_id, agent_id, probe_name, target_kind, target_ref,
  status ('pass'|'fail'|'warn'|'skip'|'error'),
  output jsonb, duration_ms int, error text,
  baseline_duration_ms int        -- rolling p50 for regression hints

monitoring_artifacts
  id, run_id, run_step_id (nullable), agent_id,
  kind ('http_response'|'screenshot'|'stack_trace'|'log_excerpt'
       |'json_blob'|'diff'|'test_scaffold'),
  storage ('inline'|'r2'),
  inline_data text (nullable), r2_key text (nullable),
  size_bytes int, content_type text,
  created_at

monitoring_findings
  id, dedup_key (unique per agent), agent_id,
  first_seen_run_id, last_seen_run_id, occurrence_count int,
  consecutive_run_count int,
  severity ('critical'|'high'|'medium'|'low'|'info'),
  title text, description text, recommended_action text,
  surface text, target_ref text,
  metadata jsonb,                 -- carries draft_task_payload etc.
  status ('open'|'acknowledged'|'dismissed'|'resolved'),
  ack_by, ack_at, ack_notes,
  resolved_at,
  created_at, updated_at

monitoring_manifest
  id, agent_id, target_kind ('route'|'table'|'edge_fn'|'package'
                            |'file'|'cron_job'|'service'),
  target_ref text, probe_config jsonb,
  source ('crawler'|'llm_merge'|'manual'|'bootstrap'),
  approval_status ('active'|'pending_approval'|'rejected'),
  added_at, added_by, last_seen_at, enabled bool

monitoring_heartbeats
  id, source ('orchestrator'|'deadman'), recorded_at, meta jsonb

monitoring_audit_log
  id, actor_id (app_users.id, nullable for system),
  action text,                    -- 'agent_model_changed'|'agent_paused'|...
  target_kind, target_id,
  before jsonb, after jsonb,
  notes text,
  created_at

Dedup keyed on (agent_id, dedup_key). Each probe computes a stable dedup key (e.g. uptime:route:/en/relations/fundraising:status_5xx). On second occurrence the existing finding's last_seen_run_id and occurrence_count increment instead of creating a new row.

Regression-aware severity. When a finding first opens, severity comes from the reasoner. When the same dedup key reopens after N successful runs (default 30), severity is bumped one level. When a probe's duration exceeds 3× its baseline_duration_ms, a performance_regression finding is auto-generated.

Retention. monitoring_run_steps, monitoring_probes, monitoring_artifacts rows older than 90d auto-delete via pg_cron job; R2 artifacts pruned in step. monitoring_runs, monitoring_findings, monitoring_audit_log retained indefinitely.

6. Reasoner & model strategy

The reasoner is the LLM execution layer. Two paths:

6.1 One-shot path (reasoner/oneshot.ts)

Direct Anthropic API call with a structured-output schema. The orchestrator collects probe results, hands them to the reasoner alongside diff context and manifest summary, and gets back a summary + findings array. No tool calling; one prompt, one response.

export interface OneShotInput {
  agent: { name: string; surface: string; description: string };
  runContext: { startedAt: Date; trigger: string; previousRunSummary?: string };
  probes: ProbeResult[];
  diffSinceLastRun?: { commits: string[]; files: string[]; patches: string };
  manifestSummary: { covered: string[]; new: string[] };
}

export interface ReasonerOutput {
  summary: string;
  findings: Array<{
    dedupKey: string;
    severity: 'critical'|'high'|'medium'|'low'|'info';
    title: string;
    description: string;
    recommendedAction: string;
    surface: string;
    targetRef?: string;
  }>;
  cost: { promptTokens: number; completionTokens: number; usd: number };
  modelUsed: string;
}

export interface OneShotReasoner { ask(input: OneShotInput): Promise<ReasonerOutput>; }

Used by: Uptime, Deploy verifier, DB integrity, Edge function health, Security active.

6.2 Agent-loop path (reasoner/agent-loop.ts)

Built on the Claude Agent SDK (@anthropic-ai/agent-sdk, same library as Claude Code). The orchestrator instantiates an SDK session per agent run with:

• The agent's system prompt (loaded from monitoring_prompt_templates)
• A tool catalog — TS functions exposed as SDK tools (see §6.3)
• Model + max-turns from agent config
• An event subscriber that writes every turn + tool call to monitoring_run_steps in real time

Used by: Security passive, Test coverage.

The SDK handles the loop (model → tool call → tool result → model → …) until the model emits no more tool calls or max_turns is reached. Final emitted findings come via the record_finding tool — the orchestrator collects these and persists them after dedup.

export interface AgentLoopRunner {
  run(input: {
    agent: AgentConfig;
    systemPrompt: string;
    tools: ToolCatalog;
    initialContext: { runId: string; diff?: GitDiff; manifest: ManifestSummary };
    maxTurns: number;
    onStep: (step: PersistableStep) => Promise<void>;  // streamed to monitoring_run_steps
  }): Promise<{ findings: Finding[]; cost: Cost; modelUsed: string; turns: number }>;
}

6.3 Tool catalog

Lives in apps/control/src/lib/sentinel/tools/. Each tool is a typed function with a Zod input schema, exposed to the SDK via the standard tool-definition shape. Tools delegate to the same probe library used by one-shot agents — so the capability is shared, only the calling style differs.

Tools execute on the orchestrator process (not on Anthropic infra) — they have direct local network access to Supabase, R2, the working tree, and CF API.

6.4 Default model + mode assignments for v1

Both mode and model are mutable per agent from the UI. Mode changes are visible in the audit log alongside model changes. The rationale field surfaces in the activity log and run drill-down so anyone reviewing a run knows why this configuration was used at the time.

Migration intent: once we have ~4 weeks of steady-state runs + cost data, evaluate which agents could downshift to a local Ollama model on alpu.ca without quality loss. The Agent SDK supports custom model providers, so the agent-loop path will accept Ollama too.

7. Orchestrator on alpu.ca

Lives at infra/alpu/sentinel-orchestrator/ (committed; deployed via rsync + systemd). Stack: Node 20 + TypeScript, runs as sentinel.service.

Schedule

Single sweep nightly inside the 10pm–8am CST window. Cron at 0 22 * * * America/Chicago kicks off the orchestrator; agents run sequentially in default order (cheapest/fastest first, most-expensive last):

Whole sweep typically finishes by ~03:30 CT. Each agent records start/end so the activity log shows actual cadence; sequencing is config-driven.

Entry points

• cron — system cron at 22:00 CT
• webhook listener on localhost:8765 for manual "run now" requests forwarded by a Cloudflare Tunnel; HMAC-signed (shared secret in ~/.config/sentinel/.env)
• heartbeat task every 15min — POSTs to a Supabase function that updates monitoring_heartbeats

Run flow per agent

1. Insert monitoring_runs row (status=running)
2. Gather manifest entries; execute probes; collect outputs + artifacts
3. Compute diff since last successful run for this agent (commits via git log, file changes)
4. Call reasoner via configured adapter; persist findings (with dedup logic)
5. Update monitoring_runs (status, cost, summary)

Auth & secrets. Bitwarden CLI (bw) unlocks at service start with a session token; service-role Supabase key + Anthropic key + R2 keys stored in ~/.config/sentinel/.env. Bootstrap recipe added to infra/runbook.md.

8. Coverage extension — crawler + LLM-on-merge

Crawler (runs at the start of each nightly sweep)

• Walks apps/garden/** and apps/control/src/app/**/page.tsx to enumerate routes
• Walks apps/control/supabase/functions/ for edge fns
• Queries information_schema.tables filtered to public schema
• Reads apps/*/package.json for dependency lists
• For each newly-seen target, inserts a monitoring_manifest row with source='crawler', approval_status='active', and a default probe config

LLM-on-merge (GitHub Action)

• Triggers on push: branches: [main]
• Reads commit range + diff
• Calls Anthropic API (Haiku) with the diff + current manifest summary
• Posts proposed manifest entries to a Supabase edge function (sentinel-llm-merge) which writes them to monitoring_manifest with approval_status='pending_approval'
• Admin sees them in Sentinel → Coverage → Pending and approves/rejects

Crawler = always-on default coverage; LLM = high-quality additions when code lands. Coverage grows automatically while the human stays in the loop on what gets monitored.

9. Frontend — Sentinel intranet section

This is where someone unfamiliar with the system has to be able to land and orient. The frontend carries the entire weight of the "dashboard only" stance.

9.1 Information architecture

Detail views are right-side drawers (mirroring the existing task-detail-drawer.tsx pattern). This keeps the static-export build workable — tab pages are pre-rendered, drawer content is fetched client-side from Supabase. Drawer state is stored in URL hash so detail views are linkable: /en/sentinel/runs#run=abc123.

9.2 Tab-by-tab spec

Overview

• Hero status card: pill (green/yellow/red), last sweep timestamp, "X agents passed / Y failed last night"
• Open critical + high findings count + a quick-list (top 3 with severity, title, surface)
• Today's activity excerpt (last 10 entries) with "View all" link
• Quick-link tiles to: All findings · All runs · Coverage matrix · Pending coverage approvals
• Polling: every 15s

Coverage

The "at-a-glance" surface map.

• Filters above the matrix: surface kind (all / routes / tables / edge fns / packages / services), agent (multi-select), status (all / has-failure / no-coverage)
• Search box for filtering by surface name
• Matrix table:
  • Rows: surfaces grouped by kind, with kind heading rows (sticky)
  • Columns: each agent that can cover this surface kind
  • Cells: small status pill (pass = green dot, fail = red, warn = amber, never run = gray, not applicable = empty)
  • Cell click → popover with last run for that probe + button "Open run drawer"
• "Surfaces without coverage" callout at the top if any surface has no agent covering it
• Sub-section below the matrix: Pending coverage approvals — manifest rows with approval_status='pending_approval' from LLM-on-merge, with Approve / Reject buttons inline

Agents

• Card grid (3 cols desktop, 1 mobile)
• Each card: agent name + surface domain icon, status pill, last run (relative time), open findings badge, 30-run sparkline (plain SVG), model chip with rationale tooltip, schedule, "Run now" button (HMAC-signed POST → orchestrator)
• Click card → opens Agent drawer with tabs:
  • About — markdown rendered from apps/control/src/components/sentinel/docs/{slug}.md. Covers purpose, surfaces covered, probes run, severity rules, default model + rationale, known limitations.
  • Probes — manifest entries this agent owns
  • Prompts — list of prompt templates with version history; click → render full prompt text + variable list
  • Runs — last 30 runs with link to run drawer
  • Findings — currently open findings for this agent
  • Settings — model + rationale (editable), cost ceiling, timeout, schedule. Edits go to monitoring_audit_log.

Activity

• Chronological feed, newest at top
• Each row: timestamp, event icon (run/finding/manifest/audit/heartbeat), actor (system / cron / user), summary, model chip + cost chip if applicable, "Open" button to relevant drawer
• Filters: event type, actor, agent, date range
• Each entry expandable to show details inline
• Live append: new entries fade in at top via 15s polling
• "Load more" pagination (50 at a time)

Findings

• Filters: severity (multi), status (open/ack/dismissed/resolved), agent (multi), surface (multi), recurrence count (≥1, ≥5, ≥30)
• Saved filter "presets": "Open critical/high", "Recurring (≥5)", "Acknowledged but not resolved", "Test coverage drafts"
• Table columns: severity badge, title, surface, agent, occurrences, last seen, status
• Sort: severity desc (default), last seen desc, recurrence desc, age asc
• Click row → opens Finding drawer
• Bulk actions toolbar: select multiple → Acknowledge / Dismiss with shared note

Finding drawer:

• Header: severity badge (large), title, status pill, surface, agent
• Description (full reasoner text)
• Recommended action (bordered box, highlighted)
• Occurrence history: list of run links with timestamps; sparkline of severity over time
• Related artifacts (collapsible viewers)
• For test-coverage findings: Approve & queue primary button + draft task preview (target file, suggested framework, prompt, scaffold). Clicking opens a confirmation modal; on confirm, inserts into tasks table and shows the task link.
• Actions footer: Acknowledge (with note), Dismiss (with reason), Resolve, View related runs
• Audit log of actions on this finding shown at bottom

Runs

• Filters: agent, trigger (cron/manual/deadman), status, date range
• Table columns: started_at, agent, trigger, status, duration, cost, models used, finding count
• Click row → opens Run drawer — the multi-step transparency surface

Run drawer:

• Header: agent name, execution-mode chip (one-shot or agent-loop), trigger + actor, status pill, total cost, total duration, models used (chips), turn count (agent-loop only)
• Stepper (vertical timeline): each step rendered as a card showing index + name, kind chip (probe / reasoner / agent_turn / tool_call / aggregate), model chip with rationale tooltip, duration + cost, status pill
• For agent-loop runs, the stepper visually groups consecutive agent_turn + child tool_call steps so it's obvious which turn invoked which tool. Tool-call cards show tool name + arguments (collapsed) and the result (collapsed).
• Card expands inline to reveal: Prompt (template name + version + collapsible rendered text — for agent-loop, this is the system prompt for the first turn and the running message history for subsequent turns), Input context (collapsible jsonb tree), Output (collapsible jsonb tree), Artifacts produced + artifacts consumed with handoff arrows
• Findings raised in this run (sidebar or bottom strip) with quick-jump to Finding drawer

Docs

System-level documentation hub. Designed for a new collaborator to onboard cold. Three subsections:

• Overview — renders infra/sentinel/README.md
• Per-agent docs — list of seven agent docs with click-to-render
• Glossary — renders infra/sentinel/glossary.md (terms: agent, probe, finding, dedup key, manifest, etc.)

All rendered with react-markdown; no MDX runtime needed.

9.3 Component catalog

Lives in apps/control/src/components/sentinel/. New shared components:

9.4 Data layer

• Single composable hook useSentinel() in apps/control/src/hooks/use-sentinel.ts
• Sub-hooks: useAgents(), useRun(runId), useRuns({ filter }), useFindings({ filter }), useFinding(id), useCoverage(), useActivity({ filter, page }), useManifest({ status })
• All read-side hooks share a small in-memory cache (15s TTL); polling tabs (Overview/Activity) refetch on interval; detail drawers refetch on open
• Mutations go through useSentinelMutations(): acknowledgeFinding, dismissFinding, resolveFinding, approveManifestEntry, rejectManifestEntry, updateAgentSettings, triggerAgentRun (HMAC-signed POST), approveAndQueueTask (creates a row in tasks from a finding's metadata.draft_task_payload)
• Every mutation invalidates relevant cached queries

9.5 Loading / empty / error

• Skeleton placeholders for all card grids and tables (one row of pulse-animated bars)
• Empty: "No findings open" — emerald-tinted card with green checkmark; "No agents have run yet" — guidance to trigger first run; "No coverage for this surface" — call-to-action to open a manifest manual entry
• Error: red-bordered card with error message + Retry button + "Report issue" link

9.6 Visual design

• Inherits intranet design: slate-on-white, Tailwind 4, no shadcn
• Severity colors: critical=rose-700, high=red-600, medium=amber-600, low=blue-500, info=slate-400
• Status colors: pass=emerald-500, fail=rose-600, warn=amber-500, skip/never=slate-300, paused=slate-400
• Surface accent colors (thin left border on agent cards / matrix column headers): uptime=blue, deploy=purple, db=teal, edge-fn=indigo, security-passive=amber, security-active=rose, test-coverage=emerald
• Typography: Inter for body, JetBrains Mono for code/IDs
• Spacing: cards 5-unit padding; section headings 6-unit margin; matrix cells dense (1.5-unit padding)

9.7 Accessibility

• Keyboard navigable: all rows/cards focusable; arrow keys navigate matrix; Enter to open drawer; Esc to close drawer
• Screen reader friendly: severity badges have aria-label; status pills include text in addition to color
• Color is never the only signal: every status/severity pairs color with text or icon

9.8 Routing

Static-export-compatible:

• /[lang]/sentinel/[tab]/page.tsx for the seven tab pages — uses <TabContent section="sentinel" /> like existing sections
• All detail views are right-side drawers; drawer state is in URL hash (e.g. #run=abc-123, #agent=uptime, #finding=xyz) so they're linkable + bookmarkable
• Drawers fetch their data client-side on open

9.9 Auth gating

• Wrap section in <AdminGuard> (new file apps/control/src/components/auth/admin-guard.tsx) checking appUser.role IN ('oracle','admin')
• Demo-role users see the section hidden from nav and a 403 if they navigate by URL
• <AdminGuard> is created in this work but not retrofitted onto the existing admin section — separate decision

10. Notifications: daily morning digest

A single Cloudflare Worker (apps/control/worker/sentinel-digest/) on a 7am CT cron. Queries last 24h of runs/findings/audit log via Supabase service role. Renders an HTML email via Resend.

Recipients: app_users with role IN ('oracle','admin').

Content:

• Headline: green/yellow/red status + count of failed agents
• New high/critical findings (title, surface, recommended action — link to Finding drawer)
• Recurring findings whose severity bumped overnight
• Cost summary: total, by agent
• Pending coverage approvals count
• Failed runs (if any) with link
• Footer: link to Sentinel dashboard

11. Files to create / modify

Create

• apps/control/migrations/20260506_sentinel_schema.sql — schema (all monitoring_* tables)
• apps/control/migrations/20260506_sentinel_rls.sql — RLS policies + admin-only access
• apps/control/migrations/20260506_sentinel_seed_prompts.sql — seed monitoring_prompt_templates
• apps/control/src/lib/sentinel/probes/{uptime,deploy,db-integrity,edge-fn,security-passive,security-active,test-coverage}.ts
• apps/control/src/lib/sentinel/reasoner/oneshot.ts — direct Anthropic API path (used by deterministic agents)
• apps/control/src/lib/sentinel/reasoner/agent-loop.ts — Claude Agent SDK runner (used by Test coverage + Security passive)
• apps/control/src/lib/sentinel/reasoner/step-recorder.ts — common step persistence; subscribes to SDK events for agent-loop, called manually for one-shot
• apps/control/src/lib/sentinel/tools/index.ts — tool registry
• apps/control/src/lib/sentinel/tools/{http-probe,query-supabase,read-file,git-diff,run-gitleaks,query-osv,inventory,find-existing-tests,draft-test-scaffold,cf-pages,r2-list,record-finding}.ts — individual tool implementations
• apps/control/src/lib/sentinel/manifest.ts — crawler logic
• apps/control/src/lib/sentinel/artifacts.ts — inline-vs-R2 storage helper
• apps/control/src/components/sentinel/{overview,coverage,agents,activity,findings,runs,docs}/ — one folder per tab
• apps/control/src/components/sentinel/shared/ — shared components
• apps/control/src/components/sentinel/docs/{uptime,deploy,db-integrity,edge-fn,security-passive,security-active,test-coverage}.md — per-agent in-UI docs
• apps/control/src/components/sentinel/drawers/{agent,run,finding,probe}-drawer.tsx
• apps/control/src/components/auth/admin-guard.tsx
• apps/control/src/hooks/use-sentinel.ts
• apps/control/src/hooks/use-sentinel-mutations.ts
• apps/control/src/app/[lang]/(intranet)/sentinel/[tab]/page.tsx
• apps/control/worker/sentinel-deadman/ — CF Worker dead-man switch
• apps/control/worker/sentinel-trigger/ — receives "Run now" UI clicks, HMAC-signs and POSTs to alpu.ca via Tunnel
• apps/control/worker/sentinel-digest/ — daily morning digest at 7am CT via Resend
• apps/control/supabase/functions/sentinel-llm-merge/ — edge function called by GitHub Action
• infra/alpu/sentinel-orchestrator/ — Node service + systemd unit + README
• infra/sentinel/README.md — system-level documentation
• infra/sentinel/glossary.md
• .github/workflows/sentinel-coverage.yml — LLM-on-merge coverage extension
• vitest.config.ts + Playwright config + sample tests per app — bootstrap delivered as a prompt-runner task in Phase 0

Modify

• apps/control/src/types/intranet.ts — add sentinel section + tabs
• apps/control/package.json — add react-markdown (UI), @anthropic-ai/agent-sdk (server-side, only used by orchestrator imports), zod for tool schemas; Vitest + Playwright added by the Phase 0 bootstrap task
• apps/garden/package.json — add Vitest + Playwright via bootstrap task
• infra/runbook.md — Sentinel orchestrator bootstrap section + alpu.ca recipe + Bitwarden recipe
• CLAUDE.md (root) — short "Monitoring" section pointing to this system
• apps/control/scripts/lint-image-gen.sh — extend whitelist for sentinel probes if they touch the image catalog

12. Build sequence — phased & sequenced

Five phases. Each item is a discrete merge-able PR. Items within a phase are mostly parallelizable; items across phases have dependencies (called out where they cross).

Phase 0 — Prerequisites (~2 days)

Goal: unblock everything else. All items must complete before Phase 2's test-coverage agent can produce useful findings.

Exit criteria: alpu.ca reachable; tunnel up; HMAC secret in place; Vitest + Playwright PR merged with at least one passing test per app; CI runs them.

Phase 1 — Foundation (~1 week, depends on Phase 0)

Goal: a single agent runs end-to-end with full step transparency.

Exit criteria: A nightly cron run on alpu.ca produces monitoring_runs + monitoring_run_steps rows for Uptime and Deploy verifier. UI Run drawer shows every step with model + rationale + prompt + input + output + artifacts. Manual "Run now" works from UI.

Phase 2 — Full agent set + Coverage view (~1.5 weeks, depends on Phase 1)

Goal: all 6 non-test agents firing nightly; Coverage matrix lit up; severity bumps and audit log working.

Exit criteria: Five of seven agents fire nightly in one-shot mode (Uptime, Deploy verifier, DB integrity, Edge function health, Security active). Findings dedupe correctly. Severity bumps after 30 successful runs. Coverage matrix is populated. Audit log captures admin actions. Cost ceiling auto-pauses an agent on test. Tool registry exists and is consumable.

Phase 3 — Agent-loop path + Coverage extension + Test agent + Notifications (~1.5 weeks, depends on Phase 2)

Goal: Claude Agent SDK wired into the orchestrator; the two agent-loop agents (Security passive, Test coverage) running; coverage auto-grows; daily digest goes out.

Exit criteria: A new route added in a feature branch shows up in monitoring_manifest after merge. Test coverage agent flags a sample uncovered file with a recommended action and a draft task; clicking Approve & queue creates a tasks row that the prompt-runner picks up. Security passive agent runs against a sample diff and emits findings via tool calls. Daily digest email arrives at 7am CT. Dead-man flagged in <25h when service stopped.

Phase 4 — Documentation & polish (~3 days, depends on Phase 3)

Goal: an unfamiliar collaborator can land in the UI and onboard cold.

Exit criteria: Docs tab renders cleanly. Visual polish complete. Accessibility audit passes. Runbook updated.

Total estimate

Phases 0–4 ≈ 4–4.5 weeks of focused work (slightly longer than the pre-SDK plan because Phase 3 now also covers the agent-loop runtime + tool catalog). Mostly sequential because the orchestrator + UI + agents stack on each other. Phase 0 should start immediately so the prompt-runner has time to deliver Vitest/Playwright bootstrap before Phase 3.

13. Verification — end-to-end checklist

1. Manual UI trigger. Click "Run now" on the Uptime agent in Sentinel → Agents. Run appears in Activity within 30s. Run drawer shows every step with model, rationale, prompt, input context, output, artifacts.
2. Nightly cron. journalctl -u sentinel.service -f on alpu.ca. Confirm 22:00 CT kickoff, sequential agent execution, full sweep complete by ~03:30 CT, exit 0.
3. Coverage matrix. Open Sentinel → Coverage. Confirm every garden + control route, every critical Supabase table, every edge function, and every external service is listed. Each row shows which agents cover it.
4. Dedup. Introduce a deliberate failure (e.g. delete a route temporarily). Run twice. Confirm finding has occurrence_count=2, not two rows.
5. Regression severity. Simulate 30 consecutive passes on a probe, then a fail; confirm severity bumps one level vs a fresh-fail finding.
6. Coverage extension. Create a branch with a new route + new edge function, merge to main. Within 2 min, GitHub Action posts manifest entries. Approve in UI. Next nightly run probes them.
7. Dead-man. Stop sentinel.service on alpu.ca. Within 25h, the dead-man Worker writes a finding with agent='deadman'. Restart service; next heartbeat resolves the finding.
8. Model swap with audit. Change Uptime agent's model_id from Haiku to Sonnet in the UI; provide a rationale. Confirm monitoring_audit_log has the change with before/after. Next run uses Sonnet and cost telemetry reflects it.
9. Cost ceiling. Lower a test agent's daily_cost_ceiling_usd to $0.001 and trigger a run. Confirm the agent auto-pauses and a finding is generated.
10. Daily digest. Confirm 7am CT email arrives at oracle/admin addresses with summary of last 24h.
11. Docs tab. Open Sentinel → Docs. Confirm README renders; each agent's doc renders; glossary visible.
12. Multi-step transparency (agent-loop). Open a Test coverage or Security passive run. Confirm the stepper shows alternating agent_turn + tool_call steps, with each tool call's name, arguments, and result visible. The execution-mode chip on the run header reads agent-loop.
13. Multi-step transparency (one-shot). Open an Uptime run. Confirm the stepper shows a flat list of probe steps followed by a single reasoner step. Execution-mode chip reads one-shot.
14. Approve & queue. Approve a test-write finding. Confirm a row appears in tasks with the draft payload; confirm the headless prompt-runner picks it up.
15. RLS. Sign in as a demo-role user ([email protected]). Confirm Sentinel tab returns 403 / hidden from nav.
16. Webhook auth. POST to the alpu.ca trigger endpoint without a valid HMAC signature; confirm rejection.

14. Out of scope for v1

• Auto-remediation (agents fixing things). Recommendations only.
• Public status page.
• Real-user metrics, Core Web Vitals, performance monitoring.
• Distributed tracing / OpenTelemetry.
• PagerDuty / on-call escalation.
• Slack notifications. (Daily morning email digest IS in v1; Slack is v1.5.)
• Multi-agent coordination / debate.
• Replacing CI's existing checks. CI stays; Sentinel sits above it.
• Auto-creation of GitHub issues from findings. (v1.5 candidate.)
• Pre-commit / pre-push hooks.
• Local-model (Ollama) reasoner adapter. Deferred to v1.1 evaluation pass once steady-state cost data is available.
• Trend analysis / week-over-week reports. (Data is captured; analytics view is v1.5.)
• Self-meta-monitoring (Sentinel observing Sentinel). Daily digest covers the basics.
• Ingesting external service status feeds (CF, Supabase status pages). v1.5.
• Retrofitting <AdminGuard> onto the existing admin section.
• Browser-based smoke tests via Playwright in Uptime. Revisit in v1.5 once Playwright is bootstrapped.
• Slow-query monitoring via pg_stat_statements.

15. Open questions to revisit during build

• Heartbeat frequency. 15min is a first guess. If monitoring runs ever need >15min uninterrupted, raise to 30min and bump dead-man window to 90min.
• Manifest approval UX. The pending_approval flow can become tedious. If approval rate is ~100% in the first month, switch the LLM-on-merge default to active and reserve pending_approval for security-touching surfaces.
• Test framework on a static-export Next.js setup. Confirm Vitest + Playwright play nicely with the next-intl-wrapped App Router. May need a small dance with the [lang] segment in test fixtures. Surface in Phase 0 task prompt.
• Sequential agent timing. The 8-step schedule assumes typical run durations; if Security passive or Test coverage routinely overruns, reorder or parallelize. Schedule is config-driven, so tunable.
• R2 cost. Artifact storage in R2 is cheap but not free. Set a per-agent artifact size cap; large bodies get truncated with a "view full body via R2" link.
• Local model picks for v1.1. Validate Qwen2.5 vs Llama 3.3 on alpu.ca with a representative probe-output sample before committing in code.

Doc owner: Haydn. Drafted 2026-05-06. Markdown source: docs/devtasks/sentinel.md. Operational secrets in BW folder devops-sponic.