Research Engineer, Synthetic Data & Evaluations

Fleet AI, Inc.; New York (NY); On-site; USD 150,000 - 190,000; Full time

Job summary

Fleet AI, Inc. is seeking a Research Engineer to build reusable systems for synthetic data and evaluations, spanning data generation, verifiers, quality measurement, and orchestration.

You will develop new methods, test on open-ended research problems, and turn successful approaches into scalable tools used by researchers. You will collaborate with internal research teams and frontier labs, study model behavior, and drive feedback loops to improve systems, with on-site work in San Francisco or

Qualifications

• Strong engineering fundamentals and experience building reliable systems that others depend on.
• Proficient in Python or another modern backend language; comfortable across containers, distributed systems, and GPUs when the work requires it.
• Strong ML fundamentals. You understand how training objectives and optimization shape model behavior, and can reason from first principles about why an approach works or fails.
• Ability to turn a research prototype into a maintainable system with clear interfaces, reliable execution, and useful debugging tools.
• Strong judgment about data and system quality: able to use expert knowledge and research signals to define what “good” looks like, translate ambiguous quality goals into useful signals, and diagnose why a system or its outputs are failing.
• Ability to work effectively with internal research teams, domain experts, and frontier lab partners.

Responsibilities

• Build reusable and reliable systems for producing high-quality synthetic data, tasks, and verifiers across software engineering, cyber, legal, finance, healthcare, and other technical and knowledge work domains.
• Leverage research signals and expert knowledge to define and enforce quality standards for each eval set—understand what “good” looks like for each domain and iterate until it is right.
• Build agentic hill-climbing and feedback loops that continuously improve the systems and their outputs against those standards.
• Collaborate closely with environment and research teams, frontier labs, and Fleet’s expert network to translate research needs and model failures into shared system capabilities.
• Treat the systems and their outputs as products: own quality and usability end to end, including researcher education, adoption, and continuous iteration based on how the systems are used.

Skills

Python

Backend development

Distributed systems

ML fundamentals

System design

Debugging

Collaboration

Tools

Containers

Docker

Job description

Fleet AI, Inc. is seeking a Research Engineer to build reusable systems for synthetic data and evaluations, spanning data generation, verifiers, quality measurement, and orchestration.

You will develop new methods, test on open-ended research problems, and turn successful approaches into scalable tools used by researchers. You will collaborate with internal research teams and frontier labs, study model behavior, and drive feedback loops to improve systems, with on-site work in San Francisco or