Founding AI Research Engineer - Medical Benchmarks & Evals

Nolla Health; New York (NY); Vor Ort; USD 180.000 - 240.000; Vollzeit

Benefits dieser Stelle

Meal stipends

Team retreats

Unlimited vacation

Health insurance

HSA/FSA eligible

Zusammenfassung

Nolla Health is seeking an engineering leader in New York City to build and maintain evals and benchmarks for our clinical AI systems. You will write code daily, design studies, and serve as the author of record on published results, in collaboration with top physicians.

This hands-on role centers on turning clinician judgment into model performance and advancing our patient-facing AI tools. You will lead the data loop with clinicians, improve the harnesses used in production, and drive

Qualifikationen

• Built eval infrastructure for LLM systems.
• Knowledge of contamination, difficulty calibration, judge bias, and reward hacking.
• Ability to write a paper or publish benchmarks.
• Experience collaborating with clinicians.

Aufgaben

• Design and maintain benchmarks to judge clinical accuracy, safety, and documentation quality.
• Build grading stacks: deterministic checks, rubric graders, and LLM judges.
• Turn real cases into eval data with clinician review, de-identification and provenance.

Jobbeschreibung

Nolla Health is seeking an engineering leader in New York City to build and maintain evals and benchmarks for our clinical AI systems. You will write code daily, design studies, and serve as the author of record on published results, in collaboration with top physicians.

This hands-on role centers on turning clinician judgment into model performance and advancing our patient-facing AI tools. You will lead the data loop with clinicians, improve the harnesses used in production, and drive