Research Engineer, Post-Training Model Evaluations

Prof Rod; San Francisco (CA); On-site; USD 500,000 - 850,000; Full time

Job summary

ProfRod is seeking a Research Engineer to evaluate post-training models in production. You will design and run large-scale evaluations, maintain evaluation coverage, and explain regressions to research teams.

The role requires strong Python and production-system skills, plus a solid foundation in statistics and experimental design. Office attendance is expected.

Qualifications

• Designing and analyzing ML evaluations at scale.
• Strong Python and production-system skills.
• Bachelor’s degree or equivalent experience; office attendance expected.

Responsibilities

• Investigate model quality during production post-training.
• Maintain evaluation coverage and explain regressions to research teams.
• Communicate findings and insights to stakeholders.

Skills

ML evaluation at scale

Python

Production systems

Statistics

Experimental design

Education

Bachelor’s degree or equivalent experience

Tools

Python

Production systems

Job description

Research Engineer, Post-Training Model Evaluations

Investigate model quality during production post-training, maintain evaluation coverage and explain regressions to research teams.

Vacancy announcement

Who this suits: Individual qualified applicant

Accepting applications has not been verified separately from the announcement.

USD 500,000–850,000/year (salary; 2026-10-05). Published annual salary range; employer determines level.

What the institution asks for

• Experience designing and analyzing ML evaluations at scale; strong Python and production-system skills.
• Statistics and experimental design are additional strengths. Bachelor’s degree or equivalent experience; office attendance expected.

ProfRod’s suggested preparation

Understand what this work involves

Practice the paired comparisons used to investigate changes in post-training evaluation results.

Predict the difference before recomputing the saved results. Name the sampling unit, interval and one limit; keep the notebook output with your memo.