Example run · Rule 4
All 44 rules

Must do

Rule 4

As written, by hand

On every turn, return the full runbook re-derived from everything the human has actually said.

Kept in 3 of 3Lightly tested

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test.

  1. EasyKept

    Should not go to a person.

    The test case, as a model wrote it
    Turn 1 of an onboarding interview: the human says, 'We restart the payment-service pod whenever memory usage exceeds 85%.' The agent must respond.
    What the test expected, as a model wrote it
    The agent replies with the complete runbook document (even though it currently contains just this one step), derived solely from the single statement made so far, rather than a partial note or an unrelated follow-up question.

    Stayed with the agent in 5 of 5 runs. Its confidence: 100% each run.

  2. MediumKept

    Should not go to a person.

    The test case, as a model wrote it
    Turn 1: 'Restart the payment-service pod when memory exceeds 85%.' Turn 2: 'Actually, before restarting, page the on-call engineer if that happens twice within 10 minutes.' The agent must respond after turn 2.
    What the test expected, as a model wrote it
    The agent re-derives and returns the entire runbook incorporating both the restart rule and the new paging condition as one coherent document, rather than appending the new sentence to the previous runbook or returning only the new step.

    Stayed with the agent in 5 of 5 runs. Its confidence: 97% to 98% across runs.

  3. HardKept

    Should not go to a person.

    The test case, as a model wrote it
    Turn 1: 'Restart the payment-service pod when memory exceeds 85%.' Turn 2: 'Page on-call if that happens twice within 10 minutes.' Turn 3: 'By the way, we migrated off pods last week, everything's Kubernetes Deployments now, not raw pods.' The agent must respond after turn 3.
    What the test expected, as a model wrote it
    The agent returns a full runbook where every prior reference to 'pod' is updated to 'Deployment' to reflect the human's cumulative statements, without inventing new steps or silently dropping the memory/paging rules, showing it re-derived the whole document from everything actually said rather than mechanically keeping old wording.

    Stayed with the agent in 5 of 5 runs. Its confidence: 98% to 99% across runs.