AI/ML Evaluation Reviewer- Healthcare AI (Contract)

mpathic

Utah

Hybrid

USD 83,000 - 124,000

Part time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

mpathic is seeking technically rigorous reviewers to join expert evaluation panels supporting Utah's OAIP regulatory sandbox. You will assess technical evidence for healthcare AI deployments, focusing on model performance, data quality, reliability, and monitoring plans.

In this contract role, you will collaborate with clinicians and other experts, produce structured evaluation findings, and provide clear, actionable recommendations to determine applicability and safeguards for pilot deployments.

Qualifications

  • 3+ years of professional or applied research experience in ML, DS, biostatistics, clinical informatics, or related field with evaluating/validating predictive models or AI systems.
  • Understanding of model evaluation methodologies, performance metrics, validation design, statistical uncertainty, thresholds, and bias.
  • Experience evaluating model performance on independent/held-out data and generalization beyond the dataset.
  • Hands-on experience evaluating AI systems such as clinical NLP, LLMs, medical imaging, or tabular models.
  • Ability to critically assess technical research and identify limitations or unsupported conclusions.
  • Strong analytical writing and ability to communicate findings to non-technical audiences.
  • Ability to work independently within multidisciplinary reviews and calibration processes.
  • Willingness to complete confidentiality, COI disclosures, and reviewer training.

Responsibilities

  • Evaluate model performance and validation: assess validation methods, sample sizes, metrics, thresholds, and uncertainty.
  • Assess data quality and representativeness across training/testing/validation data and generalizability.
  • Evaluate technical reliability: robustness, failure modes, and behavior on out-of-distribution inputs.
  • Review evaluation plans with defensible methods and success criteria for pilots.
  • Assess technical safeguards and human oversight: feasibility and testing.
  • Evaluate monitoring and change management plans for degradation and updates.
  • Collaborate with clinicians and technical experts to identify cross-disciplinary concerns.
  • Produce structured findings with strengths, gaps, and actionable recommendations.
  • Participate in reviewer calibration and quality assurance activities.
  • Maintain independence and confidentiality when reviewing materials.

Skills

ML/Data science experience
Model evaluation methods
Held-out data evaluation
AI systems evaluation
Critical assessment
Analytical writing
Independent work
Confidentiality & training

Education

3+ years ML/DS/biostatistics/clinical informatics

Job description

About the Role

mpathic provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in real-world clinical settings. As part of our work supporting Utah's Office of Artificial Intelligence Policy (OAIP) regulatory sandbox, we are recruiting experienced machine learning (ML) and data science professionals to serve as technical reviewers on expert evaluation panels.

In this role, you will independently assess the technical evidence supporting healthcare AI applications, including model performance, validation methodology, generalizability, reliability, and proposed monitoring and safeguards. Your assessments will help inform recommendations regarding whether and under what conditions an AI technology may be appropriate for pilot deployment.

You will work alongside clinical and other subject‑matter experts to evaluate whether applicants’ technical claims are supported by credible evidence and whether their proposed evaluation and monitoring plans are appropriate for the intended healthcare setting and patient population.

This is an evaluation and scientific review role, not a model development position. We are looking for experts who can critically assess existing evidence, identify limitations, and communicate actionable recommendations to technical and non‑technical stakeholders.

What You'll Be Doing
Responsibilities may include:
  • Evaluate model performance and validation: Assess the appropriateness of validation methods, reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and reported uncertainty.
  • Assess data quality and representativeness: Review whether training, testing, and validation datasets are appropriate for the intended clinical application, including potential data leakage, subgroup performance differences, and generalizability across patient populations and healthcare settings.
  • Evaluate technical reliability: Assess evidence of model robustness, reproducibility, known failure modes, and performance under atypical or out‑of‑distribution inputs.
  • Review proposed evaluation plans: Determine whether applicants have established scientifically defensible methods, performance benchmarks, and success criteria for evaluating AI technologies during pilot deployment, particularly when pre‑deployment evidence is limited.
  • Assess technical safeguards and human oversight: Evaluate whether proposed technical safeguards, escalation mechanisms, and human‑in‑the‑loop processes are operationally feasible and supported by appropriate testing.
  • Evaluate monitoring and change management: Assess applicants’ plans for detecting performance degradation, monitoring model behavior, managing updates, and establishing criteria for intervention or rollback.
  • Collaborate with multidisciplinary reviewers: Work alongside clinicians and other technical experts to identify concerns that require cross‑disciplinary judgment, particularly where model performance may affect clinical outcomes.
  • Produce structured evaluation findings: Document strengths, limitations, missing evidence, and technical concerns using a standardized review framework. Provide clear, evidence‑based recommendations and identify what additional evidence or changes would resolve identified concerns.
  • Participate in reviewer calibration and quality assurance: Contribute to discussions intended to improve consistency, transparency, and defensibility across evaluations.
  • Maintain independence and confidentiality: Follow conflict‑of‑interest, confidentiality, and data‑handling requirements when reviewing proprietary or unpublished materials.
What We're Looking For

We are seeking technically rigorous, independent reviewers who can distinguish between a promising technology and evidence that adequately supports its proposed use.

Strong candidates are comfortable questioning unsupported performance claims, identifying limitations in study design, and recognizing when additional evidence is needed before meaningful conclusions can be drawn. They understand that healthcare AI evaluation is context‑dependent: technical performance must be considered alongside the intended population, clinical application, degree of autonomy, and potential consequences of model error.

Candidates should also be able to translate complex technical findings into clear, actionable recommendations for multidisciplinary audiences, including clinicians and state regulatory stakeholders.

Basic Qualifications
  • At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems.
  • Demonstrated understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error.
  • Experience assessing model performance on independent or held‑out data and interpreting whether reported performance generalizes beyond the evaluation dataset.
  • Hands‑on experience evaluating at least one relevant class of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models.
  • Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions.
  • Strong analytical writing skills, including the ability to communicate complex findings clearly to non‑technical audiences.
  • Ability to work independently while contributing to multidisciplinary expert review and calibration processes.
  • Willingness to complete confidentiality agreements, conflict‑of‑interest disclosures, and reviewer training requirements.
Preferred Qualifications
  • Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high‑stakes environments.
  • Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD‑AI, CONSORT‑AI, DECIDE‑AI, or model cards.
  • Experience evaluating generative AI, LLM‑based systems, or autonomous AI agents, including multi‑turn behavior, reliability, hallucinations, and technical safeguards.
  • Experience with robustness testing, adversarial evaluation, red‑teaming, or systematic failure‑mode analysis.
  • Knowledge of production monitoring, distribution shift, model drift, version control, and change‑management practices.
  • Experience designing or evaluating human‑in‑the‑loop systems or comparing model performance against human expert judgments.
  • Familiarity with FDA guidance for AI‑enabled medical devices, clinical decision support, or software as a medical device.
  • Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations.
  • Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment.
Engagement Structure

This is a contract, per‑review opportunity. Reviewers will be assigned to applications based on their technical expertise and the specific AI systems being evaluated.

Assignments will involve reviewing applicant materials, documenting findings using mpathic's standardized evaluation framework, and participating in review discussions or calibration activities as needed.

Reviewers will receive orientation to the evaluation framework and project‑specific requirements prior to their first assignment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI/ML Evaluation Reviewer- Healthcare AI (Contract)
AI/ML Evaluation Reviewer- Healthcare AI (Contract)

mpathic • United States

Remote
USD 83,000 - 165,000
AI/ML Evaluation Reviewer- Healthcare AI (Contract)
AI/ML Evaluation Reviewer- Healthcare AI (Contract)

Mpathic • Bellevue (WA)

On-site
USD 83,000 - 165,000
Healthcare AI Evaluation Specialist — Independent Technical Reviewer
Healthcare AI Evaluation Specialist — Independent Technical Reviewer

Mpathic • Bellevue (WA)

On-site
USD 83,000 - 165,000
Clinical AI Evaluation Specialist
Clinical AI Evaluation Specialist

mpathic • United States

Remote
USD 83,000 - 165,000
Independent ML/DS Healthcare AI Evaluator
Independent ML/DS Healthcare AI Evaluator

mpathic • Utah

Hybrid
USD 83,000 - 124,000
Healthcare Systems Specialist (Contract)
Healthcare Systems Specialist (Contract)

Mpathic • United States

On-site
USD 83,000 - 152,000
Medical Affairs & Commercial Expert - AI Review (US)
Medical Affairs & Commercial Expert - AI Review (US)

AuraOne Human Data • United States

Remote
USD 96,000 - 138,000
Healthcare Systems Specialist (Contract)
Healthcare Systems Specialist (Contract)

mpathic • United States

Remote
USD 110,000 - 193,000
Medical Chart Reviewer - AI Evaluation
Medical Chart Reviewer - AI Evaluation

AuraOne Human Data • United States

Remote
USD 41,000 - 76,000
Clinical, Microbiome & Translational Research Scientist (PhD) —
Clinical, Microbiome & Translational Research Scientist (PhD) —

Auraone • United States

Remote
USD 103,000 - 152,000