Domain Expert, AI Evaluation Task Writer: Healthcare (Contract, Remote)

Patronus AI, Inc.; United States; Remote; USD 69,000 - 207,000; Part time

Job summary

Patronus AI, Inc. is seeking qualified clinicians and healthcare professionals to design and author realistic healthcare tasks for AI evaluation. You will create records, write task requests, solve tasks end-to-end, and define scoring criteria.

The role is remote, independent, and contract-based, with rolling start after a trial task and a commitment of 10+ hours per week. No AI background is required; you should be able to spot details in charts that affect care plans.

Qualifications

• 3+ years hands-on clinical or related work.
• Current or past licensure/certification in your field (MD/DO, NP, PA, RN, PharmD, CPC, CCS, CDIP, CCDS, CCM, CRCR).
• Knowledge of guidelines and documentation standards (ICD-10-CM/PCS, CPT, InterQual/MCG).
• Careful reading with chart-citation for every conclusion.
• Clear, precise professional English writing.
• Ability to work independently and asynchronously against deadlines.

Responsibilities

• Build fictional patient/operational records from a scenario.
• Write clear task requests with deliverables and handling of uncertainty.
• Solve tasks end-to-end and document a reference solution with reasoning.
• Write explicit scoring criteria for graders, including safe/unsafe answers.
• Run tasks against current models and adjust difficulty.
• Check accuracy and consistency of tasks and avoid shortcuts.

Skills

Years of hands-on experience

Licensure/certification

Reading and citing sources

Clear writing in professional English

Independent asynchronous work

Detail-oriented clinical observation

Education

MD/DO or NP/PA/RN/Pharmacist licensure

Specialty board credentials (CPHQ, CRCR, etc.)

Tools

Epic/Oracle Health/Solventum 360 Encompass

InterQual/MCG criteria

payer portals and claims systems

Job description

About Patronus AI

Patronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.

We are the team behind some of the earliest and most influential research in AI evaluation like FinanceBench, Lynx, SimpleSafetyTests, CopyrightCatcher, Humanity’s Last Exam, and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.

At a glance

• Engagement: Independent contractor, fully remote, asynchronous
• Pay: $50-150/hr for task authoring, paid when the full task is submitted, plus a potential bonus for each accepted task. Strong performers move up within that range or onto productivity-based pay. The trial task is paid.
• Commitment: 10+ hours per week
• Start: Rolling, as soon as you pass the trial task
• Location: Remote

The role

You’ll design realistic healthcare tasks that a qualified professional could solve with the right knowledge and tools, and that today’s AI models get wrong. Each task is built around a scenario we provide. You create the records someone in your role would actually work from, write the request, solve it yourself and define how the answer gets graded. Your work becomes evaluation and training data for frontier models.

You don’t need an AI background. You do need to be the kind of clinician or healthcare professional who spots the detail in a chart that changes the plan.

Areas we’re hiring for

• Clinical practice: physicians, nurse practitioners, physician assistants, registered nurses and pharmacists doing assessment, diagnosis, orders, medication management and care planning, from chart review and medication reconciliation through discharge instructions
• Clinical documentation and coding: clinical documentation integrity and medical coding, including ICD-10-CM/PCS and CPT/HCPCS assignment, DRG and risk adjustment, provider queries and coding audits
• Utilization management and care coordination: prior authorization and medical necessity review against InterQual or MCG criteria, level-of-care and length-of-stay decisions, clinical denials and peer-to-peer appeals, discharge and transition planning and case management
• Revenue cycle and payer operations: eligibility and benefits verification, charge capture and claims, remittance and payment posting, billing-side denials and appeals, payer policy and contract terms, and patient financial services

What you’ll do

• Build realistic, fully fictional patient and operational records from a provided scenario: clinical notes and discharge summaries, lab and imaging reports, medication lists and care plans, coding worksheets and provider queries, prior authorization requests and medical necessity criteria, utilization review notes, claims, remittances and denial letters, payer policies and eligibility responses
• Write clear task requests that state what the deliverable is and how uncertainty and justified assumptions should be handled
• Solve each task end to end and document a reference solution with the clinical or operational reasoning behind every step
• Write explicit scoring criteria that let two independent graders reach the same verdict, including acceptable alternative answers and anything that would make a response unsafe
• Run tasks against current models, find where and why they fail, and adjust difficulty
• Check your own and peers’ tasks for clinical accuracy, ambiguity, answers exposed in the materials and shortcuts that skip the intended reasoning
• Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases you run into

Who we’re looking for

Required

• 3+ years of hands-on experience in one or more of the four areas above, as a physician (MD/DO), nurse practitioner, physician assistant, registered nurse or pharmacist, or in clinical documentation, medical coding, utilization review, care management, revenue cycle or payer operations. You need to have done the work yourself, not only supervised it.
• Current or past licensure or certification in your field (for example MD/DO, NP, PA, RN, PharmD, CPC, CCS, CDIP, CCDS, CCM or CRCR)
• Working knowledge of the guidelines, criteria and documentation standards that govern your area (for example clinical practice guidelines, ICD-10-CM/PCS and CPT coding rules, InterQual or MCG criteria, CMS and payer policies)
• Careful reading, and the habit of citing the chart entry, code set or policy behind every conclusion
• Clear, precise writing in professional English
• Ability to work independently and asynchronously against deadlines

Nice to have

• Board certification or credentials beyond the minimum for your role (for example a clinical specialty board, CPHQ, CRCR or CHFP)
• Experience across settings (inpatient, outpatient, payer, pharmacy, post-acute)
• Experience with the systems used in your area, such as Epic or Oracle Health, Solventum (formerly 3M) 360 Encompass, InterQual or MCG, Availity or a payer portal, and a claims clearinghouse
• Teaching, precepting, board-question writing or clinical case development, or prior experience writing or reviewing AI evaluation tasks
• Familiarity with how AI models fail (fabricated citations, outdated guidelines, missed contraindications, wrong codes, overconfident answers)

Our standards

• All tasks, documents and solutions are human-authored. LLM-generated content is not permitted.
• Tasks and solutions must be original. Nothing may be copied from published or easily searchable sources.
• Never use real patient data or PHI, even de-identified. All patients and scenarios are fictional.
• Quality is judged on accuracy, rubric validity, difficulty and reviewer agreement, not volume.

How hiring works

1. Short AI interview and resume review
2. One paid trial task, scored against a fixed quality checklist
3. Onboarding and your first project

We aim to make a decision within two weeks of your application.

Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.