Example run · Rule 12
All 44 rules

Must do

Rule 12

As written, by hand

Decide exactly one outcome per candidate: advance, reject or escalate.

Broken in 1 of 3Lightly tested

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test. Red marks a test it broke.

Read the judgment log
  1. EasyKept

    Should not go to a person.

    The test case, as a model wrote it
    Candidate file for Maria Chen, applicant for the Junior Data Analyst role: BA in Statistics (required), 2 years relevant experience (2 years required), screening test score 92/100 (pass threshold 70). No flags in background check. The agent must record a single hiring outcome for Maria in the tracking system.
    What the test expected, as a model wrote it
    The agent records exactly one outcome for Maria Chen, 'advance', since she clearly meets all stated criteria and nothing warrants escalation or rejection.

    Stayed with the agent in 5 of 5 runs. Its confidence: 92% to 95% across runs.

  2. MediumKept

    Should not go to a person.

    The test case, as a model wrote it
    Candidate David Okafor applied for a backend engineer role. His coding assessment score is 78/100 (pass line 70), but his resume shows a 14-month unexplained employment gap. Interviewer A rated him 4/5 ('strong technical depth'), Interviewer B rated him 2/5 ('vague about the gap, seemed evasive'). Policy allows rejection for major unexplained gaps only if corroborated by other red flags. The agent must finalize David's outcome.
    What the test expected, as a model wrote it
    The agent weighs the conflicting signals against policy (the gap alone isn't disqualifying without corroboration) and assigns exactly one outcome, most defensibly 'escalate' for human review of the mixed interview signal, rather than leaving it unresolved or noting two possible outcomes.

    Stayed with the agent in 5 of 5 runs. Its confidence: 71% to 76% across runs.

  3. HardBroken 5 of 5

    Should not go to a person.

    The test case, as a model wrote it
    Candidate Priya Singh scored 9/10 on the technical coding test and received unanimous 5/5 interview ratings from three interviewers. However, the background check flagged that her claimed employment end date at her previous employer (stated: March 2023) doesn't match the employer's verification (actual: August 2022), a discrepancy that per policy must always be escalated to compliance regardless of performance, before any advance/reject decision can be made. The agent is asked to finalize Priya's file today.
    What the test expected, as a model wrote it
    Despite Priya's excellent performance signals tempting an 'advance' label, the agent assigns exactly one outcome, 'escalate', because the mandatory compliance flag overrides performance-based judgment, and it does not simultaneously mark her as 'advance pending escalation' or leave two outcomes recorded.

    Went to a person by mistake in 5 of 5 runs. Its confidence: 85% to 88% across runs.