Radiology AI Evaluation Specialist

Next Frontier Capital; United States; On-site; USD 165,000 - 276,000; Full time

Job summary

mpathic.ai is seeking experienced radiologists to help evaluate the next generation of multimodal AI systems by assessing how they interpret medical images, radiology reports, and complex clinical instructions. This benchmark dataset project focuses on evaluating AI models for visual document understanding and instruction-following in the radiology domain.

You'll develop challenging, clinically realistic evaluation tasks, create expert reference answers, and identify where today's AI systems

Qualifications

• MD, DO, or internationally equivalent medical degree.
• Completion of an accredited Diagnostic Radiology residency (ACGME, AOA, or recognized international equivalent).
• Current board certification in Diagnostic Radiology (ABR, FRCPC, FRANZCR, or recognized national equivalent).
• Active, unrestricted medical license with no suspension, revocation, or ongoing disciplinary action.
• At least 3 years of independent clinical practice following residency (training years excluded), recently within past two years.
• Experience practicing in accredited or government-licensed healthcare settings (hospitals, academic medical centers, outpatient imaging centers, or private practice).

Responsibilities

• Design clinically realistic evaluation tasks using radiology images, reports, and supporting clinical documentation.
• Create expert-level prompts that assess visual document understanding, diagnostic reasoning, and instruction-following across radiology workflows.
• Evaluate AI-generated responses using structured rubrics for diagnostic accuracy, completeness, safety, and adherence to instructions.
• Identify challenging edge cases, subtle imaging findings, and common model failure modes across multiple imaging modalities.
• Provide detailed written feedback that helps improve model performance and benchmark quality.
• Collaborate directly with research teams at a leading AI lab to advance multimodal healthcare AI.
• Participate in onboarding, calibration sessions, and ongoing quality reviews.

Skills

Unrestricted medical license

Board certification in radiology

Independent clinical practice 3+ years

Education

MD, DO, or internationally equivalent medical degree

Residency in Diagnostic Radiology (ACGME, AOA, or international equivalent)

Active board certification in Diagnostic Radiology (ABR, FRCPC, FRANZCR, or national equivalent)

Job description

mpathic.ai is seeking experienced radiologists to help evaluate the next generation of multimodal AI systems by assessing how they interpret medical images, radiology reports, and complex clinical instructions. This benchmark dataset project focuses on evaluating AI models for visual document understanding and instruction-following in the radiology domain.

You'll develop challenging, clinically realistic evaluation tasks, create expert reference answers, and identify where today's AI systems