ML Evaluation Engineer: RL Environments & Data Design

Wintermeyer Ventures; San Francisco (CA); On-site; USD 260,000 - 290,000; Full time

Job summary

Wintermeyer Ventures is seeking an applied AI researcher to join our team in San Francisco. You will build Python-based software to support reinforcement learning environments and model evaluation, helping ML systems perform more reliably in real-world domains.

You will design data collection strategies and evaluation rubrics, create data slices to surface failure modes in finance, code, and enterprise workflows, and convert nuanced evaluation needs into structured data and assessment methods

Qualifications

• 1–4 years of hands-on experience designing data collection strategies and evaluation rubrics for machine learning models.
• Experience using Python for software development.
• Background identifying model failure modes through evaluation across multiple domains, such as finance, code, or enterprise workflows.

Responsibilities

• Develop Python software that supports reinforcement learning environment work and model evaluation.
• Design data collection strategies and evaluation rubrics for machine learning models.
• Create data slices that help surface model failure modes across finance, code, and enterprise workflows.
• Help turn nuanced evaluation needs into structured data and assessment approaches that can be used to improve model performance.

Skills

Python

Data design

Reinforcement learning

Model evaluation

Job description

Wintermeyer Ventures is seeking an applied AI researcher to join our team in San Francisco. You will build Python-based software to support reinforcement learning environments and model evaluation, helping ML systems perform more reliably in real-world domains.

You will design data collection strategies and evaluation rubrics, create data slices to surface failure modes in finance, code, and enterprise workflows, and convert nuanced evaluation needs into structured data and assessment methods