Staff Engineer, Evaluation Infrastructure & ML Systems

Sierra Ventures; San Francisco (CA); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health benefits

Meals provided

Parental leave

Unlimited vacation

Visa sponsorship

Team events

Job summary

Reflection is building an evaluation infrastructure to help researchers understand model capabilities, guide training, and assess readiness for release. You will collaborate with researchers to speed up evaluations, ensure reliability, and provide user-friendly interfaces for inspecting model outputs and experiments.

The role focuses on pre-training evaluations, scalability, observability, and reproducibility, with opportunities to impact open foundational models and contribute to a globally

Qualifications

• Strong Python/C++/Rust skills and ability to reason about concurrency and memory behavior.
• Experience designing, building, and operating reliable distributed systems.
• Working knowledge of model inference and evaluation.
• Ability to investigate complex problems across system boundaries and find root causes.
• Strong product judgment translating researchers' needs into clear APIs and tools.
• Comfort owning ambiguous problems end to end while balancing speed and reliability.

Responsibilities

• Design and evolve evaluation abstractions with clear APIs and data models.
• Build scalable, reliable execution infrastructure for evaluating model checkpoints.
• Optimize evaluation and inference performance to improve throughput and time to results.
• Establish evaluation correctness and reproducibility with tests and versioning.
• Build researcher-facing tools and dashboards to inspect outputs and compare experiments.
• Operate and monitor evaluations during training, triage failures, and develop alerts.

Skills

Python

C++

Rust

Distributed systems

Concurrency

Performance

API design

Job description

Reflection is building an evaluation infrastructure to help researchers understand model capabilities, guide training, and assess readiness for release. You will collaborate with researchers to speed up evaluations, ensure reliability, and provide user-friendly interfaces for inspecting model outputs and experiments.

The role focuses on pre-training evaluations, scalability, observability, and reproducibility, with opportunities to impact open foundational models and contribute to a globally