ML Engineer: Annotation Pipelines & Evaluation

RustLabs; Connecticut; On-site; USD 120,000 - 165,000; Full time

Job summary

RustLabs is seeking an ML engineer to own core data pipelines and evaluation systems that turn raw tasker output into model-ready training data. This hands-on IC role involves designing annotation schemas with customers, building scalable tooling, and delivering data-quality at scale for frontier AI research.

You will work closely with research teams, ship production ML systems, and contribute to research artifacts.

Qualifications

• 2+ years of professional experience as an ML engineer, research engineer, or similar role.
• Shipped ML systems to production and understand data quality in practice.
• Familiarity with RLHF, SFT, or evaluation methodologies is a strong plus.
• Ability to write clean code, move fast, and talk directly to customers.
• Bonus: open-source contributions or prior AI lab experience.

Responsibilities

• Design annotation pipelines (RLHF, SFT, eval, red-teaming) end-to-end, including schema design and tasker UX.
• Build evaluation infrastructure: automated checks, LLM-as-judge systems, calibration tooling, and inter‑annotator metrics.
• Collaborate with research teams at AI labs to translate data needs into deliverable annotation products.
• Develop internal tooling to enable a distributed tasker workforce to produce high-quality data at scale.
• Contribute to research artifacts (datasets, evals, papers) when relevant work ships.

Skills

Python

ML engineer

Data quality

Customer communication

Production ML

Tools

PyTorch

HuggingFace

Job description

RustLabs is seeking an ML engineer to own core data pipelines and evaluation systems that turn raw tasker output into model-ready training data. This hands-on IC role involves designing annotation schemas with customers, building scalable tooling, and delivering data-quality at scale for frontier AI research.

You will work closely with research teams, ship production ML systems, and contribute to research artifacts.