The 30‑Day AI Observability Sprint for UK SMEs

If your AI assistant, triage bot or copilot is moving from pilot to “always on” in January, the single best gift you can give your team is a lightweight observability routine. Not a new tool; a routine: clear KPIs, sensible alert thresholds, and a weekly review that drives decisions. This article lays out a practical 30‑day sprint UK SMEs and charities can run to make AI features dependable in production.

Why now? Providers publish status pages, but those don’t always reflect customer‑specific issues. For example, Azure distinguishes between its public Status page and the personalised Service Health feed for your subscriptions, which is where most actionable notices appear. Relying on the public page alone will miss incidents that only affect your tenancy or region. See Microsoft’s overview of Azure Service Health and the note on when items do (and don’t) appear on the public status page. Azure Service Health, Azure status vs Service Health.

Similarly, AWS offers an account‑aware Health Dashboard alongside its public service health page, and OpenAI provides a live status page plus published rate‑limit tiers that can constrain throughput during peaks. If you operate AI in production, you should monitor all three layers: your app, your cloud, and your AI providers. AWS Health Dashboard, OpenAI Status, OpenAI rate limits.

The KPI set that fits on one slide

Keep it simple. Aim for 10–12 measures across Reliability, Quality, Cost and Usage. These are plain‑English, exec‑friendly and map cleanly to actions.

Reliability

• Success rate (2xx) and provider errors (4xx/5xx) split by provider.
• Latency: p50 and p95 end‑to‑end; time‑to‑first‑token for streamed responses.
• Rate‑limit hits and quota utilisation by provider/model (e.g., tokens per minute). Azure OpenAI quotas, Amazon Bedrock quotas.
• External dependency health (OpenAI/Azure/AWS status) linked to incidents.

Quality

• Groundedness/faithfulness of answers (0–1 scale) sampled daily.
• Answer relevance (does it actually address the user’s question?).
• Context recall/precision for RAG: are we retrieving the right sources?

These quality metrics are widely used in enterprise RAG evaluations; NVIDIA’s documentation summarises faithfulness, answer relevance/correctness and context precision/recall with clear definitions. NVIDIA NeMo RAG metrics.

Cost

• Cost per resolved conversation/task (including human handovers).
• Tokens per interaction (input vs output) and daily spend burn‑down.

Usage

• Active users/sessions; containment rate (solved without handover).
• Human‑in‑the‑loop: handover rate and the top 5 handover reasons.

The 30‑day sprint (week by week)

Week 1 — Baseline and ownership

1. Nominate owners: one product owner, one ops owner, one quality owner (can be part‑time roles in SMEs).
2. Define SLOs and an error budget: pick one availability SLO (e.g., 99.5% “good” interactions over 28 days) and one latency SLO (p95 under X seconds). Error budgets let you balance speed vs stability; if you burn the budget, slow changes until you recover. Error budget policy (Google SRE).
3. Map dependencies: list each model/provider, region, and feature that depends on them. Capture the relevant status feeds: OpenAI, Azure Service Health, AWS Health.
4. Instrument the KPIs: even a spreadsheet works for a month. Use your vendor billing and logs to populate success rates, latency, token usage, and spend. Track rate‑limit events; providers enforce RPM/TPM limits which can throttle throughput at peaks. OpenAI rate limits, Azure quotas.
5. Agree initial thresholds: for example:
   • p95 latency warning at +30% above baseline for 15 minutes; critical at +60%.
   • Success rate warning at 98.5%; critical at 97% sustained 10 minutes.
   • Rate‑limit hits warning when any model exceeds 5% of calls; critical at 10%.
   • Faithfulness below 0.9 on a 50‑sample daily review triggers remediation.

For a deeper dive on stabilising operations, see our earlier post on runbooks, SLOs and on‑call.

Week 2 — Alerts, dashboards, and the review cadence

1. Send alerts to where people live: Teams/Slack channel for warnings; page only critical breaches. Tie each alert to an owner and a runbook step.
2. Set the weekly 30‑minute review: trend the KPIs, review incidents, and decide one action per domain (Reliability, Quality, Cost). Keep minutes.
3. Introduce a “brownout” mode: when upstream status pages report incidents or you’re burning error budget fast, switch to conservative settings (cheaper models, shorter context, disable non‑essential tools) to preserve service.

Week 3 — Quality loops and “test like a user”

1. Daily samples: review 50 anonymised sessions. Score faithfulness and answer relevance; tag the top 3 failure reasons (bad retrieval, unclear prompt, provider timeout).
2. Evaluate your RAG: track context precision and recall; irrelevant or missing context is the fastest path to poor answers. The NVIDIA summary of RAG metrics is a good reference. RAG metrics.
3. Fix one cause per week: e.g., shrink chunk size, improve document metadata, or add a simple “I don’t know” guard when contexts are weak.

Week 4 — Drill, incident hygiene and pre‑January go‑live

1. Run a 90‑minute incident drill: simulate a provider rate‑limit or regional outage. Practice your escalation and communications. UK public‑sector guidance recommends pre‑built incident playbooks; SMEs benefit from the same discipline. Incident response playbook (GOV.UK).
2. Adopt an error‑budget rule: if a single incident consumes >20% of the monthly budget, run a short post‑mortem and reserve time for fixes before shipping new features. Example policy.
3. Agree a change window: for the first two weeks of January, bundle changes and use small canaries with the ability to roll back quickly. See our guide to safe AI change management.

If you expect January peaks, pair this with a short load test and capacity plan.

Alert thresholds that won’t fry your on‑call

Borrow the SRE principle: “If it’s important enough to wake a human, it must require immediate action.” Convert that into three alert classes:

• Page Critical breach of SLO or error budget burn rate > 5% per hour; sustained provider errors; security‑sensitive anomalies. SRE alerting guidance.
• Ticket Cost per resolution up 20% week‑on‑week; answer relevance trending down; token usage spikes that don’t affect users yet.
• Log Single transient timeouts, minor model retries, small fluctuations in tokens per minute.

Note: vendor limits can change. For example, cloud AI quotas are often tiered by spend or subscription type and differ by region. Keep a simple sheet of your agreed RPM/TPM for each model and region so ops and product have the same picture. Azure OpenAI quotas, Bedrock quotas, OpenAI tiers.

Procurement questions for observability (ask vendors and your MSP)

• What rate limits and quotas apply to our current plan? Can we pre‑approve burst allowances for January?
• Do you provide account‑specific service health webhooks or emails when our resources are impacted? (Azure, AWS and others do.)
• Which quality metrics can your platform export natively (faithfulness, relevance, context precision/recall), and how are they calculated?
• What is your recommended fallback model/region when the primary is degraded? Is failover automatic, manual, or unsupported?
• How do we access billing and token usage in near real time to monitor cost per resolution?
• Do you publish post‑incident reviews for customer‑visible incidents, and how quickly? (Azure and AWS document how status and Service Health differ.)

Your weekly agenda (print this)

1. 5 minutes: Incidents since last week, status vs account‑specific health updates.
2. 10 minutes: KPIs trend. Are we within SLOs? Are alerts the right level?
3. 10 minutes: Quality loop. Review 10 examples; note top failure cause.
4. 5 minutes: One action per domain (Reliability, Quality, Cost). Assign owner and due date.

When to freeze change

Use a simple, transparent policy tied to your error budget. If the monthly budget is exhausted (or a single incident burns >20%), pause non‑urgent changes and focus on reliability work until the trend improves. Error budgets are a proven way to balance shipping speed with stability. Error budget policy.

If you need to keep shipping during busy periods, reduce blast radius: enable canaries, set strict rollback criteria, and keep rollouts during staffed hours. Pair this with our 7‑day AI reliability sprint if you’re starting from scratch.

Lightweight incident hygiene

• Keep a single “AI incidents” log with timestamps, symptoms, suspected cause, fix, and follow‑up.
• Do 30‑minute post‑incident reviews with one concrete action each time.
• Publish a short stakeholder note for significant incidents (what happened, what we changed, what to expect). UK government guidance encourages playbooks and clear communications as part of good incident practice. Response and recovery planning.

What good looks like by Day 30

• KPIs visible to product and ops, with two weeks of trends.
• Alerts are quiet most days; when they fire, someone knows exactly what to do.
• You have a working “brownout” mode and a documented fallback model/region.
• Your weekly review is a habit—and you can show improvements in latency, faithfulness and cost per resolution.

From here, you can deepen technical telemetry (logs, metrics, traces) if needed—many teams call this “Observability 2.0”, correlating signals and tying them to business outcomes—but the routine above gets you dependable service without new tools. Observability 2.0 (CNCF).