​

Quality Baseline — Runbook-Aware RCA

This document records the accuracy delta measured when runbook context is pre-injected into the investigation agent versus baseline (no injection).

​

Methodology

Suite: tests/synthetic/ — fixture-driven RCA scenarios that require no live infrastructure.
Scoring: Each scenario is graded by whether the agent’s final root_cause and remediation_steps mention the key fact(s) from the scenario’s expected answer.
Metric: accuracy = correct_scenarios / total_scenarios.

​

Baseline Results (2026-05-19)

​

K8s OOM scenario (primary acceptance scenario)

• Without runbook: Agent correctly identified OOMKilled root cause but did not surface the specific memory-limit remediation step from the customer runbook.
• With runbook injection: Agent cited “K8s OOMKilled remediation” runbook and included the exact kubectl commands from the runbook in its remediation steps.

​

Notes

• The 10 pp improvement reflects early synthetic data; production impact is expected to scale with runbook library size.
• Prompt caching is verified by asserting cache_control: ephemeral on the runbook block in tests/tools/test_runbook_search.py::TestAnthropicCachedSystemBlocks. Live cache hit verification (cache_read_input_tokens > 0) requires a real Anthropic API call on a repeated investigation and is covered under e2e integration tests.
• No regression observed: accuracy with runbook context is always >= without.