> ## Documentation Index
> Fetch the complete documentation index at: https://docs.healops.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Quality baseline

# Quality Baseline — Runbook-Aware RCA

This document records the accuracy delta measured when runbook context is
pre-injected into the investigation agent versus baseline (no injection).

## Methodology

**Suite**: `tests/synthetic/` — fixture-driven RCA scenarios that require no live infrastructure.\
**Scoring**: Each scenario is graded by whether the agent's final `root_cause` and
`remediation_steps` mention the key fact(s) from the scenario's expected answer.\
**Metric**: `accuracy = correct_scenarios / total_scenarios`.

## Baseline Results (2026-05-19)

| Condition               | Correct / Total | Accuracy   |
| ----------------------- | --------------- | ---------- |
| Without runbook context | 8 / 10          | 80%        |
| With runbook context    | 9 / 10          | 90%        |
| **Delta**               | +1              | **+10 pp** |

### K8s OOM scenario (primary acceptance scenario)

* **Without runbook**: Agent correctly identified OOMKilled root cause but did not
  surface the specific memory-limit remediation step from the customer runbook.
* **With runbook injection**: Agent cited "K8s OOMKilled remediation" runbook and
  included the exact `kubectl` commands from the runbook in its remediation steps.

## Notes

* The 10 pp improvement reflects early synthetic data; production impact is expected
  to scale with runbook library size.
* Prompt caching is verified by asserting `cache_control: ephemeral` on the runbook
  block in `tests/tools/test_runbook_search.py::TestAnthropicCachedSystemBlocks`.
  Live cache hit verification (`cache_read_input_tokens > 0`) requires a real Anthropic
  API call on a repeated investigation and is covered under e2e integration tests.
* No regression observed: accuracy with runbook context is always >= without.