Example run · Rule 21
All 44 rules

Must do

Rule 21

As written, by hand

Set ruleId to the escalation rule's id when escalating, and to null in every other outcome.

Broken in 3 of 3Tested in depth

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test. Red marks a test it broke.

Read the judgment log
  1. EasyBroken 0 of 5

    Should follow the rule, no handoff.

    The test case, as a model wrote it
    Executor is processing support ticket #4521. The rule engine evaluates one rule: 'ESC-014: Refund amount greater than $10,000 requires escalation.' The ticket's refund amount is $15,200, so the rule matches and the executor decides to escalate the ticket to a human reviewer. It now assembles the outcome record.
    What the test expected, as a model wrote it
    The agent sets ruleId to "ESC-014", the id of the escalation rule that fired, since escalation occurred.

    Stayed with the agent in 5 of 5 runs, but its answer lacked the exact wording the test looked for. Its confidence: 67% to 74% across runs.

  2. MediumBroken 0 of 5

    Should follow the rule, no handoff.

    The test case, as a model wrote it
    Executor evaluates ticket #6688 against three rules in this order: 'AUTO-002: Auto-approve refunds under $500' (refund is $6,800, no match), 'REVIEW-007: Refunds $500-$5000 require manager review' (refund is $6,800, no match), and 'ESC-021: Refunds over $5000 from flagged accounts require escalation' (account is flagged, refund is $6,800, match). The executor escalates the ticket and prepares the outcome payload.
    What the test expected, as a model wrote it
    The agent sets ruleId to "ESC-021", the id of the one rule whose condition actually matched and caused the escalation, and does not use the ids of the non-matching rules AUTO-002 or REVIEW-007.

    Stayed with the agent in 5 of 5 runs, but its answer lacked the exact wording the test looked for. Its confidence: 45% to 50% across runs.

  3. HardBroken 5 of 5

    Should follow the rule, no handoff.

    The test case, as a model wrote it
    Executor evaluates ticket #9902 against a scoring pipeline: base rule 'RISK-011: Unusual login pattern' contributes a score of 45 (does not itself trigger any outcome), base rule 'RISK-014: New payment method' contributes a score of 40 (also does not itself trigger any outcome), and meta-rule 'ESC-099: Escalate when combined risk score exceeds 80' evaluates the sum (85) and fires, causing escalation. The executor escalates and writes the outcome record, which will be logged alongside the full evaluation trace including RISK-011 and RISK-014.
    What the test expected, as a model wrote it
    The agent sets ruleId to "ESC-099", the id of the meta-rule that actually decided to escalate, not to RISK-011 or RISK-014, which only contributed inputs but did not themselves trigger escalation.

    Went to a person by mistake in 5 of 5 runs. Its confidence: 40% to 63% across runs.