Staff Engineer, ML Evaluation Infrastructure

reflectionai; New York (NY); On-site; USD 140,000 - 210,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave

Unlimited vacation (US)

Visa sponsorship

Job summary

Reflection AI is seeking a software-focused engineer to build the evaluation infrastructure used by researchers to understand model capabilities, guide training decisions, and assess readiness for release. The role spans evaluation authoring, distributed execution, and results analysis, with a focus on pre-training.

You will collaborate with researchers to design scalable APIs, robust orchestration, and intuitive dashboards that reveal model outputs, track experiments, and help distinguish

Qualifications

• Proficiency in Python, C++, or Rust.
• Experience designing, building and operating reliable distributed systems.
• Knowledge of model inference and evaluation and how config choices affect quality.
• Ability to translate research needs into clear APIs and reusable tools.

Responsibilities

• Design and evolve evaluation abstractions with clear APIs and data models.
• Build scalable, reliable distributed evaluation infrastructure and orchestration.
• Optimize evaluation and inference performance and throughput.
• Ensure reproducibility with validation, regression tests, and traceability.
• Develop researcher-facing interfaces and dashboards for inspecting outputs.
• Monitor evaluation workloads, triage failures and diagnose issues.
• Turn research needs into reusable, self-service tools across teams.

Skills

Python

C++/Rust

Distributed systems

Performance optimization

Job description

Reflection AI is seeking a software-focused engineer to build the evaluation infrastructure used by researchers to understand model capabilities, guide training decisions, and assess readiness for release. The role spans evaluation authoring, distributed execution, and results analysis, with a focus on pre-training.

You will collaborate with researchers to design scalable APIs, robust orchestration, and intuitive dashboards that reveal model outputs, track experiments, and help distinguish