Staff Engineer, Evaluation Infrastructure

Reflection; San Francisco (CA); On-site; USD 140,000 - 200,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health insurance

Meals provided in office

Parental leave program

Unlimited PTO (US)

Job summary

Reflection is seeking an engineer to build the evaluation infrastructure researchers use to understand model capabilities and readiness for release. You will work with researchers to make evaluations faster, more reliable, and easier to use, spanning authoring, distributed execution, and results analysis with emphasis on pre-training.

You will design APIs, build scalable execution systems, optimize performance, ensure reproducibility, and create researcher-facing tools and dashboards to inspect

Qualifications

• Strong Python/C++/Rust proficiency with an eye for concurrency and performance.
• Experience designing, building, and operating reliable distributed systems.
• Knowledge of model inference/evaluation workflows and scalable evaluation tooling.

Responsibilities

• Design and evolve evaluation abstractions with clear APIs and data models.
• Build scalable, reliable execution infrastructure for evaluating model checkpoints.
• Profile and optimize inference servers and distributed execution for throughput and latency.
• Ensure evaluation correctness through validation and reproducibility practices.
• Develop researcher-facing dashboards and interfaces for inspecting results.
• Monitor workloads and triage failures, distinguishing model quality changes from infra issues.
• Collaborate with researchers to turn bespoke workflows into reusable tools.

Skills

Python

C++

Rust

Distributed systems

API design

Performance tuning

Problem solving

Job description

Reflection is seeking an engineer to build the evaluation infrastructure researchers use to understand model capabilities and readiness for release. You will work with researchers to make evaluations faster, more reliable, and easier to use, spanning authoring, distributed execution, and results analysis with emphasis on pre-training.

You will design APIs, build scalable execution systems, optimize performance, ensure reproducibility, and create researcher-facing tools and dashboards to inspect