Member of Technical Staff, Evaluation

GenMD

Oregon

Hybrid

USD 150,000 - 210,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

GenMD, a Stanford spin-off building AI and data infrastructure for healthcare, seeks a research-focused engineer to design and maintain clinical benchmarks and evaluation systems for real-world healthcare data. You will run end-to-end evaluation pipelines, create rubrics and automated graders, and work with clinical experts to translate results into publications and products.

Requires hands-on LLM evaluation experience, strong Python and data engineering skills, and the ability to overlap

Qualifications

  • Hands-on experience evaluating or training LLMs.
  • Strong Python and data engineering skills with large tabular and text datasets.
  • Research mindset to design meaningful evaluations and avoid shortcuts.
  • Bachelor’s degree or higher in CS, ML, NLP, biomedical informatics; LLM/NLP publications preferred.

Responsibilities

  • Design and maintain clinical benchmarks for reasoning, longitudinal histories, multimodal records and tool use.
  • Build end-to-end evaluation pipelines: runs, scoring, dashboards and reports at scale.
  • Create rubrics, automated graders and LLM-as-a-judge systems, calibrated against clinician review.
  • Run systematic failure analysis: categorize failure modes and measure occurrence frequency.
  • Collaborate with clinicians and partner AI teams to turn benchmark results into research papers and products.

Skills

LLM evaluation
Python
Data engineering
Research mindset
BigQuery/SQL

Education

B.Sc. or higher in CS/ML/NLP/biomedical informatics

Tools

PyTorch
Hugging Face
SQL/BigQuery

Job description

About GenMD

GenMD is a Stanford spin-off building AI and data infrastructure for healthcare. We grew out of years of work inside Stanford Medicine, alongside world-class researchers and clinicians, and we now work with health systems and data partners on large longitudinal datasets: claims, health records, imaging and clinical notes. We make that data usable for researchers and frontier AI, while patient privacy stays protected. We're just coming out of stealth, already revenue-generating and well funded. We're intentionally small, move fast, and are focused on building something fundamental. Our CEO is a former Associate Director of AI at Stanford Medicine. See how we work:

About the role

Frontier models still fail on real clinical work, and most of those failures never show up in public benchmarks. You'll build the benchmarks and evaluation systems that show where models break on real-world healthcare data, then turn those findings into the data that closes the gap. We call the loop measure, build, re-test.

What you'll do
  • Design and maintain clinical benchmarks for reasoning, longitudinal patient histories, multimodal records and tool use.
  • Build evaluation pipelines end to end: runs, scoring, dashboards and reports that compare models and training runs at scale.
  • Create rubrics, automated graders and LLM-as-a-judge systems, and calibrate them against clinician review.
  • Run systematic failure analysis: categorize failure modes, measure how often they occur and feed the results into dataset design and post-training.
  • Work with clinical experts and partner AI teams to turn benchmark results into research papers and products.
What we're looking for
  • Hands-on experience evaluating or training LLMs (PyTorch, Hugging Face or similar).
  • Strong Python and data engineering skills; comfortable with large tabular and text datasets (SQL/BigQuery a plus).
  • Research taste: you can design an evaluation that measures what matters and resists shortcuts.
  • B.Sc. or higher in CS, ML, NLP, biomedical informatics or a related field; LLM/NLP publications preferred.
  • About 4 hours of daily overlap with Pacific Time (roughly 9 AM to 1 PM PT).
Nice to have
  • Experience with clinical data (EHR, claims, FHIR/OMOP, clinical notes) or medical benchmarks.
  • Experience with RL from verifiable rewards, reward modeling or post-training data.
Compensation

$150,000 – $210,000 per year, plus equity. Final offer depends on experience and location.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff Intern, Applied AI
Member of Technical Staff Intern, Applied AI

GenMD • Iowa

On-site
USD 62,000 - 83,000
Member of Technical Staff, Agentic Systems
Member of Technical Staff, Agentic Systems

GenMD • Iowa

On-site
USD 180,000 - 250,000
Equity
Senior Research Engineer, AI Evaluation
Senior Research Engineer, AI Evaluation

system • United States

Hybrid
USD 180,000 - 240,000
Equity
Health and wellness coverage
Paid time off
+1
Clinical AI Benchmark Architect
Clinical AI Benchmark Architect

GenMD • Oregon

Hybrid
USD 150,000 - 210,000
Medical Expert Consultant
Medical Expert Consultant

GenMD • Oregon

Hybrid
USD 138,000 - 241,000
Software Engineer, Health Evaluations job at Microsoft in New York, NY
Software Engineer, Health Evaluations job at Microsoft in New York, NY

Microsoft Corp. • New York (NY)

Hybrid
USD 143,000 - 275,000
Director of Machine Learning (Healthcare AI)
Director of Machine Learning (Healthcare AI)

Nxt Level • United States

Hybrid
USD 150,000 - 200,000
Competitive salary
Meaningful equity
Direct line to CEO
+1
AI/ML Engineer
AI/ML Engineer

Storm3 • Tennessee

On-site
USD 180,000 - 280,000
AI Engineer
AI Engineer

Harnham • San Francisco (CA)

On-site
USD 100,000 - 150,000
Founding Forward Deployed Machine Learning Engineer [33151]
Founding Forward Deployed Machine Learning Engineer [33151]

Stealth Startup • Sunnyvale (CA)

On-site
USD 160,000 - 220,000
0.5-2.0% Equity
Insurance