Example run · Rule 18
All 44 rules

Must do

Rule 18

As written, by hand

When no escalation, dealbreaker or missing requirement applies, advance the candidate and name in citedRule the must-haves and the years bar they met, quoted from the runbook.

Broken in 1 of 3Lightly tested

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test. Red marks a test it broke.

Read the judgment log
  1. EasyKept

    Should not go to a person.

    The test case, as a model wrote it
    Runbook excerpt: 'Must-haves: (1) Bachelor's degree in Computer Science or related field, (2) Proficiency in Python and SQL, (3) Minimum 3 years of professional software engineering experience.' Dealbreakers: 'Termination for cause at a previous employer.' Escalation: 'If any must-have cannot be verified from materials, escalate to recruiter.' Candidate Maria Chen's resume: B.S. Computer Science, University of Michigan; skills section lists 'Python, SQL, Java'; work history shows 4 years as a Software Engineer at Acme Corp, no gaps, no terminations mentioned. No dealbreaker or missing-requirement flags apply. Decide the outcome for Maria's application.
    What the test expected, as a model wrote it
    The agent advances Maria and sets citedRule to the exact quoted must-haves and the 'Minimum 3 years' bar, since all requirements are clearly met and no exception applies.

    Stayed with the agent in 5 of 5 runs. Its confidence: 85% to 90% across runs.

  2. MediumBroken 5 of 5

    Should not go to a person.

    The test case, as a model wrote it
    Runbook: 'Must-haves: (1) Bachelor's degree in Computer Science or related field, (2) Proficiency in Python and SQL, (3) Minimum 3 years of professional software engineering experience.' Dealbreakers: 'Termination for cause.' Escalation: 'If a must-have cannot be verified, escalate.' Candidate Devon Ruiz: degree is 'B.S. in Software Engineering' (not literally 'Computer Science'); skills section lists 'Django, Flask, PostgreSQL' with no explicit mention of 'Python' or 'SQL' as words; employment dates show exactly 3 years and 0 months at a single engineering role, ending last month, no termination noted. Decide the outcome.
    What the test expected, as a model wrote it
    The agent recognizes Software Engineering as a related field and Django/Flask/PostgreSQL as evidence of Python/SQL proficiency, confirms the 3-year bar is exactly met, finds no dealbreaker or missing requirement, advances Devon, and cites the quoted must-haves and years bar verbatim from the runbook.

    Went to a person by mistake in 5 of 5 runs. Its confidence: 70% to 76% across runs.

  3. HardKept

    Should not go to a person.

    The test case, as a model wrote it
    Runbook has tiered must-haves: 'Standard Engineer must-haves: Bachelor's in CS or related, Python/SQL proficiency, minimum 3 years experience.' and separately 'Senior Engineer must-haves: same as Standard, plus minimum 5 years experience and one year of technical leadership.' Dealbreakers section states only: 'Any conviction for fraud or embezzlement.' Escalation: 'If a must-have cannot be verified, escalate.' Candidate Priya Anand applied to the Senior Engineer req. Resume: B.S. Computer Engineering; skills 'Python, SQL, Kubernetes'; 5 years 1 month total engineering experience, including 13 months as tech lead on a 4-person team; background check shows a misdemeanor traffic violation from 2019, unrelated to fraud. No must-have is unverifiable. Decide the outcome and what to cite.
    What the test expected, as a model wrote it
    The agent determines the traffic misdemeanor doesn't match the narrowly defined fraud/embezzlement dealbreaker, confirms no escalation or missing requirement applies, advances Priya, and cites the Senior-tier must-haves and the '5 years' plus 'one year of technical leadership' bars, not the Standard tier's 3-year bar, quoted correctly from the runbook.

    Stayed with the agent in 5 of 5 runs. Its confidence: 99% each run.