Example run · Rule 28
All 44 rules

Must do

Rule 28

As written, by hand

Extract niceToHaves as the preferred-but-not-required signals.

Broken in 3 of 3Tested in depth

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test. Red marks a test it broke.

Read the judgment log
  1. EasyBroken 0 of 5

    Should follow the rule, no handoff.

    The test case, as a model wrote it
    Extract the structured requirements from this job posting: "Backend Engineer Requirements: - 5+ years of Python experience - Experience with REST APIs Nice to have: - Experience with Docker - Familiarity with AWS"
    What the test expected, as a model wrote it
    The agent outputs niceToHaves: ["Experience with Docker", "Familiarity with AWS"], keeping Python experience and REST APIs experience in mustHaves rather than mixing them in.

    Stayed with the agent in 5 of 5 runs, but its answer lacked the exact wording the test looked for. Its confidence: 99% to 100% across runs.

  2. MediumBroken 0 of 5

    Should follow the rule, no handoff.

    The test case, as a model wrote it
    Extract the structured requirements from this job posting: "We're looking for a backend engineer. Must have 3+ years Node.js. Experience with GraphQL is a strong plus. Should be comfortable with Git. Bonus points for contributions to open source projects. Required: bachelor's degree in CS or equivalent experience."
    What the test expected, as a model wrote it
    The agent outputs niceToHaves: ["Experience with GraphQL", "Contributions to open source projects"], while placing Node.js experience, Git comfort, and the degree requirement in mustHaves, since no explicit section header separates them and the agent must parse hedge words like 'plus' and 'bonus'.

    Stayed with the agent in 5 of 5 runs, but its answer lacked the exact wording the test looked for. Its confidence: 99% each run.

  3. HardBroken 0 of 5

    Should follow the rule, no handoff.

    The test case, as a model wrote it
    Extract the structured requirements from this job posting dated 12 May 2024: "Candidates must have a valid engineering degree and at least 4 years in backend development. Experience with Kubernetes is preferred but not mandatory; candidates lacking it may still be considered if they demonstrate strong Docker skills during the interview. Knowledge of Terraform is a definite plus, though it is not evaluated in the initial screening."
    What the test expected, as a model wrote it
    The agent outputs niceToHaves: ["Experience with Kubernetes", "Knowledge of Terraform"] only, correctly excluding 'strong Docker skills' since that phrase names an interview-stage fallback condition rather than a standalone preferred signal, and keeping degree and 4+ years backend experience in mustHaves.

    Stayed with the agent in 5 of 5 runs, but its answer lacked the exact wording the test looked for. Its confidence: 99% to 100% across runs.