Synthetic Data & Evals

Fleet AI, Inc.; New York (NY); On-site; USD 150,000 - 190,000; Full time

Job summary

Fleet AI, Inc. is seeking a Research Engineer to build reusable systems for synthetic data and evaluations, spanning data generation, verifiers, quality measurement, and orchestration.

You will develop new methods, test on open-ended research problems, and turn successful approaches into scalable tools used by researchers. You will collaborate with internal research teams and frontier labs, study model behavior, and drive feedback loops to improve systems, with on-site work in San Francisco or

Qualifications

• Strong engineering fundamentals and experience building reliable systems that others depend on.
• Proficient in Python or another modern backend language; comfortable across containers, distributed systems, and GPUs when the work requires it.
• Strong ML fundamentals. You understand how training objectives and optimization shape model behavior, and can reason from first principles about why an approach works or fails.
• Ability to turn a research prototype into a maintainable system with clear interfaces, reliable execution, and useful debugging tools.
• Strong judgment about data and system quality: able to use expert knowledge and research signals to define what “good” looks like, translate ambiguous quality goals into useful signals, and diagnose why a system or its outputs are failing.
• Ability to work effectively with internal research teams, domain experts, and frontier lab partners.

Responsibilities

• Build reusable and reliable systems for producing high-quality synthetic data, tasks, and verifiers across software engineering, cyber, legal, finance, healthcare, and other technical and knowledge work domains.
• Leverage research signals and expert knowledge to define and enforce quality standards for each eval set—understand what “good” looks like for each domain and iterate until it is right.
• Build agentic hill-climbing and feedback loops that continuously improve the systems and their outputs against those standards.
• Collaborate closely with environment and research teams, frontier labs, and Fleet’s expert network to translate research needs and model failures into shared system capabilities.
• Treat the systems and their outputs as products: own quality and usability end to end, including researcher education, adoption, and continuous iteration based on how the systems are used.

Skills

Python

Backend development

Distributed systems

ML fundamentals

System design

Debugging

Collaboration

Tools

Containers

Docker

Job description

Research Engineer, Synthetic Data & Evals

About Fleet

Fleet studies how environments produce intelligence. We believe intelligence is an emergent property of environmental pressures — the environment determines what capabilities develop, what behaviors survive, and what "good" looks like. We work with frontier labs on post-training across modalities — building benchmarks that expose where frontier models break, training recipes that close those gaps, and scalable oversight for long-horizon agents. Backed by Sequoia Capital, Menlo Ventures, BCV, and SV Angel.

About the Role

We’re looking for a Research Engineer to build reusable and reliable systems for synthetic data and evaluations.

This work spans synthetic data and task generation, verifiers, quality measurement, and the systems that connect and orchestrate the entire loop. You will develop new methods, test them on open-ended research problems, and turn the ones that work into systems that you and other researchers can use and extend.

You will work closely with internal research teams and frontier labs, study model behavior and traces directly, and build feedback loops between research findings and system improvements.

Success means making high-quality research faster, more reproducible, and easier to scale across researchers and domains, with the longer-term goal of systems that use evaluation feedback to recursively improve their own outputs.

What You'll Work On

• Build reusable and reliable systems for producing high-quality synthetic data, tasks, and verifiers across software engineering, cyber, legal, finance, healthcare, and other technical and knowledge work domains.
• Leverage research signals and expert knowledge to define and enforce quality standards for each eval set—understand what “good” looks like for each domain and iterate until it is right.
• Build agentic hill-climbing and feedback loops that continuously improve the systems and their outputs against those standards.
• Collaborate closely with environment and research teams, frontier labs, and Fleet’s expert network to translate research needs and model failures into shared system capabilities.
• Treat the systems and their outputs as products: own quality and usability end to end, including researcher education, adoption, and continuous iteration based on how the systems are used.

What We're Looking For

• Strong engineering fundamentals and experience building reliable systems that others depend on. Proficient in Python or another modern backend language; comfortable across containers, distributed systems, and GPUs when the work requires it.
• Strong ML fundamentals. You understand how training objectives and optimization shape model behavior, and can reason from first principles about why an approach works or fails.
• Ability to turn a research prototype into a maintainable system with clear interfaces, reliable execution, and useful debugging tools.
• Strong judgment about data and system quality: able to use expert knowledge and research signals to define what “good” looks like, translate ambiguous quality goals into useful signals, and diagnose why a system or its outputs are failing.
• Ability to work effectively with internal research teams, domain experts, and frontier lab partners.
• High ownership, low ego, and a preference for systems that are simple and boring.

We care more about demonstrated ability than credentials. Strong candidates may come from research engineering, backend or data systems, ML platforms, developer tooling, synthetic data, or applied AI.

Especially Exciting

• Post-training experience and research taste. You’ve owned experiments to improve model capabilities, designed ablations to understand what worked, and can identify promising research directions while distinguishing meaningful progress from misleading results.
• Hands-on experience building synthetic or domain-specific datasets and eval sets, reward models, agentic workflows, or automated quality systems.

How We Work

• Small, technical team.
• Speed with rigor.
• Truth over comfort. Honest with partners, with each other, and with ourselves.
• On-site in SF or NYC.

Location

On-site in San Francisco or New York. Highly competitive salary + equity.