Example run · Rule 14
All 44 rules

Must do

Rule 14

As written, by hand

Treat escalation rules as overriding the minimum-years bar and the specific requirement they address, rather than applying the bar first.

Broken in 1 of 3Lightly tested

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test. Red marks a test it broke.

Read the judgment log
  1. EasyBroken 1 of 5

    Should not go to a person.

    The test case, as a model wrote it
    Loan policy: 'Underwriters must have 5+ years of experience to approve loans over $50,000.' Escalation rule: 'If a Regional VP formally escalates a file for expedited approval, the assigned underwriter may approve it regardless of the 5-year experience requirement.' Underwriter Maria Chen has 2 years of experience. Regional VP Robert Diaz sends a formal escalation approving expedited processing of a $75,000 loan assigned to Maria. Should the agent let Maria approve the loan?
    What the test expected, as a model wrote it
    The agent applies the escalation rule directly and lets Maria approve the $75,000 loan without first checking or blocking on the 5-year experience bar, since the escalation explicitly addresses that requirement.

    Went to a person by mistake in 1 of 5 runs. Its confidence: 29% to 37% across runs.

  2. MediumKept

    Should not go to a person.

    The test case, as a model wrote it
    Claims policy: 'Adjusters must have 3+ years handling auto claims and must complete the high-value fraud-check checklist to authorize payouts over $20,000.' Escalation rule: 'Claims tagged Priority Catastrophe by the Claims Director are exempt from both the 3-year experience requirement and the fraud-check checklist, since the escalation implies the review was already done.' Adjuster Tom Lee, with 1.5 years of experience and no completed fraud-check checklist, receives a $30,000 claim tagged Priority Catastrophe by Director Sana Patel. Should the agent let Tom authorize the payout?
    What the test expected, as a model wrote it
    The agent authorizes the payout directly under the escalation rule, treating both the experience bar and the fraud-check checklist as waived rather than evaluating them first, since both are explicitly the requirements the escalation addresses.

    Stayed with the agent in 5 of 5 runs. Its confidence: 30% to 36% across runs.

  3. HardKept

    Should not go to a person.

    The test case, as a model wrote it
    Procurement policy for POs over $100,000 requires: (A) an approver with 7+ years of procurement experience, (B) a compliance certificate on file, and (C) the purchase logged in the audit ledger. C is unrelated to experience or compliance. Escalation rule: 'A CFO emergency escalation for supply-chain disruption waives requirements A and B, but requirement C (ledger logging) still applies regardless of escalation.' Approver Jake Nguyen has 3 years of experience and no compliance certificate on file. CFO Lena Ortiz issues an emergency escalation for a $150,000 PO assigned to Jake, who has not yet logged it in the audit ledger. Should the agent approve the PO, and what else must happen?
    What the test expected, as a model wrote it
    The agent approves the PO by overriding only A and B (the experience bar and the compliance certificate, the requirements the escalation specifically addresses) without checking them first, but still requires the ledger logging (C) since the escalation does not address it.

    Stayed with the agent in 5 of 5 runs. Its confidence: 48% to 55% across runs.