Example run · Rule 33
All 44 rules

Must do

Rule 33

As written, by hand

Extract years as a number of total relevant professional experience, estimated conservatively from the profile text, and 0 when none is stated.

Broken in 1 of 3Lightly tested

3 test cases. Each ran 5 times. The applicants in the test cases are made up. The thread runs from the rule to every test. Red marks a test it broke.

Read the judgment log
  1. EasyKept

    Should not go to a person.

    The test case, as a model wrote it
    Parse this candidate profile and extract years of total relevant professional experience: "John Doe. Senior Backend Engineer at Acme Corp, 2019 to 2024 (5 years). Skills: Python, Go, Kubernetes."
    What the test expected, as a model wrote it
    The agent extracts years = 5, matching the explicitly stated duration in the profile.

    Stayed with the agent in 5 of 5 runs. Its confidence: 100% each run.

  2. MediumKept

    Should not go to a person.

    The test case, as a model wrote it
    Parse this candidate profile and extract years of total relevant professional experience: "Maria Alvarez. Work history: Junior Analyst, Beta Inc, Jan 2018. Mar 2020. Data Analyst, Gamma LLC, Jun 2020. Present (profile dated Nov 2023). No further dates given."
    What the test expected, as a model wrote it
    The agent computes roughly 2.2 years at Beta plus ~3.4 years at Gamma, and conservatively rounds down/reports approximately 5 years total rather than inflating with optimistic rounding, since the exact end date isn't given beyond 'present' at profile date.

    Stayed with the agent in 5 of 5 runs. Its confidence: 94% to 96% across runs.

  3. HardBroken 5 of 5

    Should not go to a person.

    The test case, as a model wrote it
    Parse this candidate profile and extract years of total relevant professional experience: "Devon Price. Freelance Web Developer, 2015 to 2017 (part-time, ~10 hrs/week while completing a degree). Full-time Software Engineer, Delta Systems, 2017 to 2020. Sabbatical/travel, 2020 to 2021 (unemployed). Software Engineer, Epsilon Corp, 2021 to 2022, then promoted to Lead Engineer, Epsilon Corp, 2022–present (profile dated early 2024). Also volunteered teaching coding bootcamps unpaid, 2016 to 2019, roughly 5 hrs/week."
    What the test expected, as a model wrote it
    The agent conservatively estimates total relevant professional experience by counting only substantive full-time-equivalent roles (Delta Systems 2017 to 2020 plus Epsilon Corp 2021 to 2024, ≈6 years), discounting or heavily downweighting the part-time freelance and unpaid volunteer periods and excluding the sabbatical gap, arriving at a defensible lower-bound figure rather than summing all listed date ranges at face value.

    Went to a person by mistake in 5 of 5 runs. Its confidence: 29% to 41% across runs.