Independent ML/DS Healthcare AI Evaluator

Mpathic; Utah; Hybrid; USD 83,000 - 124,000; Part time

Job summary

mpathic is seeking technically rigorous reviewers to join expert evaluation panels supporting Utah's OAIP regulatory sandbox. You will assess technical evidence for healthcare AI deployments, focusing on model performance, data quality, reliability, and monitoring plans.

In this contract role, you will collaborate with clinicians and other experts, produce structured evaluation findings, and provide clear, actionable recommendations to determine applicability and safeguards for pilot deployments.

Qualifications

• 3+ years of professional or applied research experience in ML, DS, biostatistics, clinical informatics, or related field with evaluating/validating predictive models or AI systems.
• Understanding of model evaluation methodologies, performance metrics, validation design, statistical uncertainty, thresholds, and bias.
• Experience evaluating model performance on independent/held-out data and generalization beyond the dataset.
• Hands-on experience evaluating AI systems such as clinical NLP, LLMs, medical imaging, or tabular models.
• Ability to critically assess technical research and identify limitations or unsupported conclusions.
• Strong analytical writing and ability to communicate findings to non-technical audiences.
• Ability to work independently within multidisciplinary reviews and calibration processes.
• Willingness to complete confidentiality, COI disclosures, and reviewer training.

Responsibilities

• Evaluate model performance and validation: assess validation methods, sample sizes, metrics, thresholds, and uncertainty.
• Assess data quality and representativeness across training/testing/validation data and generalizability.
• Evaluate technical reliability: robustness, failure modes, and behavior on out-of-distribution inputs.
• Review evaluation plans with defensible methods and success criteria for pilots.
• Assess technical safeguards and human oversight: feasibility and testing.
• Evaluate monitoring and change management plans for degradation and updates.
• Collaborate with clinicians and technical experts to identify cross-disciplinary concerns.
• Produce structured findings with strengths, gaps, and actionable recommendations.
• Participate in reviewer calibration and quality assurance activities.
• Maintain independence and confidentiality when reviewing materials.

Skills

ML/Data science experience

Model evaluation methods

Held-out data evaluation

AI systems evaluation

Critical assessment

Analytical writing

Independent work

Confidentiality & training

Education

3+ years ML/DS/biostatistics/clinical informatics

Job description

mpathic is seeking technically rigorous reviewers to join expert evaluation panels supporting Utah's OAIP regulatory sandbox. You will assess technical evidence for healthcare AI deployments, focusing on model performance, data quality, reliability, and monitoring plans.

In this contract role, you will collaborate with clinicians and other experts, produce structured evaluation findings, and provide clear, actionable recommendations to determine applicability and safeguards for pilot deployments.