Staff Engineer, Healthcare AI Benchmarks

GenMD; United States; Remote; USD 150,000 - 210,000; Full time

Job summary

GenMD, a Stanford spin-off, builds AI and data infrastructure for healthcare. You’ll design clinical benchmarks for reasoning and longitudinal histories, and build evaluation pipelines that run at scale to compare models and training runs on real-world data.

Collaborate with clinical experts and partner AI teams to turn benchmark results into research papers and products. We value practical judgment, strong Python/data skills, and a bias for rigorous evaluation.

Qualifications

• Hands-on experience evaluating or training LLMs (PyTorch, Hugging Face or similar).
• Strong Python and data engineering skills; comfortable with large tabular and text datasets (SQL/BigQuery a plus).
• Research taste: you can design an evaluation that measures what matters and resists shortcuts.
• B.Sc. or higher in CS, ML, NLP, biomedical informatics or a related field; LLM/NLP publications preferred.
• About 4 hours of daily overlap with Pacific Time (roughly 9 AM to 1 PM PT).

Responsibilities

• Design and maintain clinical benchmarks for reasoning, longitudinal patient histories, multimodal records and tool use.
• Build evaluation pipelines end to end: runs, scoring, dashboards and reports that compare models and training runs at scale.
• Create rubrics, automated graders and LLM-as-a-judge systems, and calibrate them against clinician review.
• Run systematic failure analysis: categorize failure modes, measure how often they occur and feed the results into dataset design and post-training.
• Work with clinical experts and partner AI teams to turn benchmark results into research papers and products.

Skills

LLM evaluation/training

Python & data engineering

Research-minded evaluation

Timezone overlap

Education

B.Sc. or higher in CS/ML/NLP/biomedical informatics

Tools

SQL/BigQuery

Job description

GenMD, a Stanford spin-off, builds AI and data infrastructure for healthcare. You’ll design clinical benchmarks for reasoning and longitudinal histories, and build evaluation pipelines that run at scale to compare models and training runs on real-world data.

Collaborate with clinical experts and partner AI teams to turn benchmark results into research papers and products. We value practical judgment, strong Python/data skills, and a bias for rigorous evaluation.