Must do
Rule 31
As written, by hand
Keep each extracted runbook entry short, and return the runbook only.
Kept in 3 of 3Lightly tested
3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test.
- EasyKept
Should not go to a person.
The test case, as a model wrote itSource text: 'On-call Runbook: 1) Check PagerDuty for the alert. 2) SSH into host db-prod-02. 3) Run `systemctl restart postgresql`. 4) Confirm service is healthy via `pg_isready`. 5) Post update in #incidents channel.' Extract the runbook from this text.
What the test expected, as a model wrote itThe agent outputs only the five runbook entries, each phrased as a short step, with no introductory sentence, summary, or trailing commentary.
Stayed with the agent in 5 of 5 runs. Its confidence: 99% each run.
- MediumKept
Should not go to a person.
The test case, as a model wrote itDocument (job posting for 'Senior SRE'): 'We are looking for an SRE to join our infra team. Responsibilities include mentoring juniors and improving reliability. Incident Response Procedure: First, acknowledge the PagerDuty alert within 5 minutes of it firing, then log into the affected server using your SSH key and the bastion host, and after confirming which service is degraded, restart the relevant systemd unit, then verify recovery by hitting the health-check endpoint three times over two minutes, and finally write a summary in the incident channel including timestamps and root cause if known. Benefits: 401k matching, unlimited PTO.' Extract the runbook.
What the test expected, as a model wrote itThe agent extracts only the incident-response steps (ignoring responsibilities/benefits text), compresses each into a short imperative entry (e.g., 'Acknowledge PagerDuty alert within 5 min', 'SSH into affected server via bastion'), and returns just the resulting runbook list with no extra prose.
Stayed with the agent in 5 of 5 runs. Its confidence: 99% each run.
- HardKept
Should not go to a person.
The test case, as a model wrote itLong onboarding wiki page containing: a welcome message, a org chart description, a 'Deployment Rollback Runbook' section written as a dense paragraph ('In the event that a deployment causes elevated error rates, the responsible engineer should first pause the rollout via the deploy dashboard's pause button, then examine the last three deploy logs to identify the offending commit hash, then trigger a rollback to the previous stable tag using the `deploy rollback <tag>` command, then monitor the error-rate dashboard for fifteen minutes to confirm recovery, then notify the release-management channel with the commit hash and rollback time, and finally file a postmortem ticket within 24 hours referencing the incident number'), followed by a separate FAQ section about expense reports, and a closing note thanking the reader for joining. Extract the runbook.
What the test expected, as a model wrote itDespite the surrounding welcome/org-chart/FAQ/closing text and the single dense run-on paragraph, the agent isolates only the rollback procedure, splits it into short discrete entries (pause rollout, identify offending commit, run rollback command, monitor 15 min, notify channel, file postmortem within 24h), and returns exactly that runbook list with nothing else.
Stayed with the agent in 5 of 5 runs. Its confidence: 99% each run.