QA READY — Daryle AI DevOps Harness
What changed: The bot had no automated way to detect when it
stops responding — failures only appeared via user complaints.
Now a daily cron actually tests the bot, model errors get
logged, and models can be swapped from the admin panel without
a deploy.
What to test:
- 4-provider status cards
Before: Only OpenAI and Pinecone had status cards
After: Anthropic and Gemini cards also visible
Steps: Admin → Service Health. Count cards at top — 4
expected. Click "Run Check Now" — all 4 update.
- LLM Failure Events panel
Before: No visibility into AI model errors
After: Panel shows errors grouped by type; empty state
when system is healthy
Steps: Admin → Service Health → scroll below donut charts.
"LLM Failure Events — Last 24h" section visible.
- Model Configuration panel
Before: Changing AI model = code deploy
After: Change model from admin, live within 60 seconds
Steps: Admin → Service Health → scroll to bottom. Six rows
(grounded/fast/claude/pro/chatgpt/gemini) with
editable fields and Save buttons.
- Chat modal model info
Before: Chat modal showed messages only
After: Footer shows model name, response time, fallback
Steps: Admin → Chats → open any chat. Footer line shows
model and latency (only for chats with log data).
- Daily canary (deploy-gated)
Before: No automated health check
After: GitHub Actions fires 6am + 6pm UTC, tests each
route, emails alert on 2nd consecutive failure
Steps: GitHub → Actions → Health Canary → Run workflow.
Edge case: Requires SUPABASE_FUNCTIONS_URL +
SUPABASE_SERVICE_ROLE_KEY in repo secrets first.