Example run · Rule 9
All 44 rules

Must do

Rule 9

As written, by hand

Assign escalation rule ids R1, R2, ... in the order the human gives them.

Kept in 3 of 3Lightly tested

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test.

  1. EasyKept

    Should not go to a person.

    The test case, as a model wrote it
    Human: "Let's set up escalation rules. First, if a ticket sits untouched for 4 hours, notify the on-call lead. Second, if a customer replies 'urgent' twice, page the manager. Third, if a VIP account opens a ticket, alert the account team immediately." Assign the rule IDs.
    What the test expected, as a model wrote it
    The agent assigns R1 to the 4-hour rule, R2 to the 'urgent' twice rule, and R3 to the VIP alert rule, matching the order the human stated them.

    Stayed with the agent in 5 of 5 runs. Its confidence: 92% to 93% across runs.

  2. MediumKept

    Should not go to a person.

    The test case, as a model wrote it
    Human: "Rule for SLA breach after 24 hours goes to team lead. Oh wait, before that one, I should have mentioned: if the customer is a Tier-1 partner, escalate to the VP within 1 hour regardless of SLA. And then after the SLA one, add a rule that repeated failed payments over $10,000 get flagged to finance." Assign the rule IDs.
    What the test expected, as a model wrote it
    The agent must recognize the human's insertion language ('before that one') as reordering intent and assign R1 to the Tier-1 partner rule, R2 to the SLA breach rule, and R3 to the finance flag rule, i.e. the order the human ultimately specified, not the literal utterance order.

    Stayed with the agent in 5 of 5 runs. Its confidence: 93% to 95% across runs.

  3. HardKept

    Should not go to a person.

    The test case, as a model wrote it
    Turn 1 - Human: "Rule: if response time exceeds 2 hours, escalate to shift supervisor." Turn 2 - Human: "Also add: if a ticket is reopened 3+ times, escalate to QA." Turn 3 - Human: "Actually, I want to prioritize the reopened-ticket rule higher in importance than the response-time one, but don't change the order you gave them, just note the priority separately." Turn 4 - Human: "One more: if a ticket mentions legal threat, escalate to legal team immediately." Assign the rule IDs.
    What the test expected, as a model wrote it
    Despite the human explicitly discussing 'priority' as a separate concept, the agent assigns R1 to response-time, R2 to reopened-ticket, and R3 to legal-threat, strictly preserving the chronological order in which each rule was given, not reordering by the stated priority.

    Stayed with the agent in 5 of 5 runs. Its confidence: 91% to 93% across runs.