Clinical AI Benchmark Architect

GenMD

Oregon

Hybrid

USD 150,000 - 210,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

GenMD, a Stanford spin-off building AI and data infrastructure for healthcare, seeks a research-focused engineer to design and maintain clinical benchmarks and evaluation systems for real-world healthcare data. You will run end-to-end evaluation pipelines, create rubrics and automated graders, and work with clinical experts to translate results into publications and products.

Requires hands-on LLM evaluation experience, strong Python and data engineering skills, and the ability to overlap

Qualifications

  • Hands-on experience evaluating or training LLMs.
  • Strong Python and data engineering skills with large tabular and text datasets.
  • Research mindset to design meaningful evaluations and avoid shortcuts.
  • Bachelor’s degree or higher in CS, ML, NLP, biomedical informatics; LLM/NLP publications preferred.

Responsibilities

  • Design and maintain clinical benchmarks for reasoning, longitudinal histories, multimodal records and tool use.
  • Build end-to-end evaluation pipelines: runs, scoring, dashboards and reports at scale.
  • Create rubrics, automated graders and LLM-as-a-judge systems, calibrated against clinician review.
  • Run systematic failure analysis: categorize failure modes and measure occurrence frequency.
  • Collaborate with clinicians and partner AI teams to turn benchmark results into research papers and products.

Skills

LLM evaluation
Python
Data engineering
Research mindset
BigQuery/SQL

Education

B.Sc. or higher in CS/ML/NLP/biomedical informatics

Tools

PyTorch
Hugging Face
SQL/BigQuery

Job description

GenMD, a Stanford spin-off building AI and data infrastructure for healthcare, seeks a research-focused engineer to design and maintain clinical benchmarks and evaluation systems for real-world healthcare data. You will run end-to-end evaluation pipelines, create rubrics and automated graders, and work with clinical experts to translate results into publications and products.

Requires hands-on LLM evaluation experience, strong Python and data engineering skills, and the ability to overlap

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff, Evaluation
Member of Technical Staff, Evaluation

GenMD • Oregon

Hybrid
USD 150,000 - 210,000
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs
GenAI Evaluation Scientist — Benchmark & Diagnose LLMs

Scale AI, Inc. • New York (NY)

On-site
USD 181,000 - 226,000
Base salary + equity
Health, dental, vision coverage
Retirement benefits
+3
Senior AI Evaluation Engineer - Remote LLM Benchmarking
Senior AI Evaluation Engineer - Remote LLM Benchmarking

System • New York (NY)

Hybrid
USD 180,000 - 240,000
Meaningful equity ownership
Comprehensive health coverage
Generous paid time off
+1
Clinical AI Monitoring & Evaluation Engineer
Clinical AI Monitoring & Evaluation Engineer

Stanford University • Palo Alto (CA)

Hybrid
USD 120,000 - 180,000
Senior AI Evaluation & Benchmark Engineer (Hybrid)
Senior AI Evaluation & Benchmark Engineer (Hybrid)

Glean • San Francisco (CA)

Hybrid
USD 200,000 - 280,000
Medical, Vision, Dental
Generous time-off
401k plan
+4
Clinical AI Reviewer & Grading Expert — Remote
Clinical AI Reviewer & Grading Expert — Remote

GenMD • Oregon

Hybrid
USD 138,000 - 241,000
Remote AI Evaluation Lead - LLM Benchmarks & Safety
Remote AI Evaluation Lead - LLM Benchmarks & Safety

system • United States

Hybrid
USD 180,000 - 240,000
Equity
Health and wellness coverage
Paid time off
+1
Senior Science & Clinical AI Benchmarking Expert
Senior Science & Clinical AI Benchmarking Expert

Braintrust • United States

Remote
USD 55,000 - 77,000
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics
GenAI Evaluation Scientist: LLM Benchmarks & Diagnostics

Scale AI, Inc. • San Francisco (CA)

On-site
USD 166,000 - 207,000
Health coverage
Equity
Retirement benefits
+3
Staff Research Scientist, AI Benchmarking & Evaluation
Staff Research Scientist, AI Benchmarking & Evaluation

Thomas Talent Network, LLC • San Francisco (CA)

On-site
USD 140,000 - 210,000