Senior Research Engineer, AI Evaluation

system

United States

Hybrid

USD 180,000 - 240,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity
Health and wellness coverage
Paid time off
Flexible work arrangement

Job summary

system is seeking an engineer/researcher to lead the design and implementation of evaluation systems for language models used in clinical decision support. You will combine applied research and engineering to measure model reasoning, retrieval, and structured responses, and ensure quality and safety checks are reproducible across development and deployment.

You will build evaluation frameworks, develop benchmarks, test cases, and reference datasets, and address edge cases, hallucinations, drift,

Qualifications

  • Three+ years in software engineering, applied machine learning, or AI research with strong statistics.
  • Experience building LLM evaluation harnesses, benchmarks, or reference metrics.
  • Strong technical writing and communication skills.

Responsibilities

  • Build and maintain automated evaluation frameworks for model outputs.
  • Develop general and domain-specific benchmarks, synthetic test cases, and trusted reference datasets.
  • Measure edge cases, hallucinations, model drift, and safety regressions.
  • Integrate evaluation practices into CI/CD workflows with research, infra, and product partners.

Skills

Python
Statistics
Scientific rigor

Education

Advanced degree in computer science / machine learning / statistics

Tools

PyTorch
Hugging Face
Model APIs
Retrieval-augmented generation
Experiment tracking / observability tools

Job description

Role overview

Lead the design and implementation of evaluation systems for language models used in clinical decision support applications. The role combines applied research and engineering to measure model reasoning, retrieval, and structured responses, and to make quality and safety checks reproducible across development and deployment.

Responsibilities
  • Build and maintain automated evaluation frameworks for model outputs.
  • Develop general and domain-specific benchmarks, synthetic test cases, and trusted reference datasets.
  • Measure and address edge cases, hallucinations, model drift, and safety regressions.
  • Establish offline and online evaluation practices with research, infrastructure, and product partners, including integration into CI/CD workflows.
Requirements
  • At least three years in software engineering, applied machine learning, or AI research, with strong statistics and scientific rigor.
  • Demonstrated experience building LLM evaluation harnesses or benchmarks, such as model-based judging, reference metrics, or human review.
  • Advanced Python skills and experience with PyTorch or Hugging Face, model APIs, retrieval-augmented generation, and experiment tracking or observability tools.
  • Strong technical writing and communication skills.
Nice to have
  • Advanced degree in computer science, machine learning, statistics, or another scientific field.
  • Peer-reviewed research or presentations at machine learning conferences.
  • Knowledge graph experience.
Benefits and work setup
  • Base salary range of $180,000–$240,000, equity, health and wellness coverage, and paid time off.
  • Flexible work arrangement with a Lower Manhattan hub and support for remote work.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Engineer, Evaluation Tooling
Software Engineer, Evaluation Tooling

Modulus • United States

Remote
USD 140,000 - 220,000
Health insurance
Paid time off
Member of Technical Staff, Evaluation
Member of Technical Staff, Evaluation

GenMD • Oregon

Hybrid
USD 150,000 - 210,000
Applied Research Engineer
Applied Research Engineer

Ragas • United States

Remote
USD 60,000 - 200,000
Remote role
Senior Backend Engineer, AI Evaluations
Senior Backend Engineer, AI Evaluations

Ellipsis Health • San Francisco (CA)

Hybrid
USD 160,000 - 210,000
401(k) matching
Health insurance
Flexible PTO
Machine Learning Engineer, Evaluation & Reliability
Machine Learning Engineer, Evaluation & Reliability

Bask Health • Oregon

Hybrid
USD 120,000 - 180,000
Lead Research Engineer - AI Evaluation & Benchmarking
Lead Research Engineer - AI Evaluation & Benchmarking

Thomson Reuters • Eagan (MN)

On-site
USD 150,000 - 210,000
Senior Software Engineer – AI Evaluation
Senior Software Engineer – AI Evaluation

Alignerr • United States

Remote
USD 83,000 - 165,000
Machine Learning Engineer, Evaluation & Reliability
Machine Learning Engineer, Evaluation & Reliability

Bask Health • United States

Remote
USD 140,000 - 200,000
AI Evaluations Engineer
AI Evaluations Engineer

Ellipsis Health • San Francisco (CA)

Hybrid
USD 160,000 - 210,000
401(k) matching
Health insurance
Flex time off
Machine Learning Engineer, Evaluation & Reliability
Machine Learning Engineer, Evaluation & Reliability

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000