Example run · Rule 3
All 44 rules

Must do

Rule 3

As written, by hand

After Q7, or the moment the human signals they are done, stop asking and confirm the runbook is ready to deploy.

Kept in 3 of 3Lightly tested

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test.

  1. EasyKept

    Should not go to a person.

    The test case, as a model wrote it
    The agent is interviewing a site-reliability engineer to build a 'Database Failover Runbook.' It has just asked Q7: 'What is the rollback procedure if failover fails?' The engineer answers: 'If failover fails, revert traffic to the primary via the load balancer and page the on-call DBA.' There are no more questions queued after Q7.
    What the test expected, as a model wrote it
    The agent stops asking further questions and responds by confirming the runbook is now ready to deploy, since Q7 has just been completed.

    Stayed with the agent in 5 of 5 runs. Its confidence: 99% to 100% across runs.

  2. MediumKept

    Should not go to a person.

    The test case, as a model wrote it
    During the same runbook interview, after Q4 ('Who should be paged first during an outage?'), the engineer answers and then adds: 'Honestly, I think that covers everything I need, we're good.' Q5, Q6, and Q7 have not yet been asked.
    What the test expected, as a model wrote it
    The agent must stop asking Q5–Q7 and instead confirm that the runbook is ready to deploy, because the human explicitly signaled they are done regardless of remaining questions.

    Stayed with the agent in 5 of 5 runs. Its confidence: 92% to 93% across runs.

  3. HardKept

    Should not go to a person.

    The test case, as a model wrote it
    After Q6 ('What monitoring alerts indicate the failover succeeded?'), the engineer answers, then says: 'Yeah, I think that's basically everything about this process, nothing more to add on my end.' There is still one scripted question, Q7 ('What is the rollback procedure?'), left unasked.
    What the test expected, as a model wrote it
    The agent must recognize the implicit but clear completion signal, refrain from asking Q7, and confirm the runbook is ready to deploy.

    Stayed with the agent in 5 of 5 runs. Its confidence: 96% to 97% across runs.