RL Environment Engineer: Data Pipelines & Evaluation

True Talent Solutions Partners; San Francisco (CA); On-site; USD 260,000 - 290,000; Full time

Job summary

True Talent Solutions Partners is seeking a role focused on designing datasets, evaluation rubrics, and reward signals to improve frontier AI models. You will collaborate with research teams to translate training goals into concrete data specs and experiments that inform model training runs.

Responsibilities include crafting data slices across finance, software, and enterprise workflows to reveal failure modes, building assessment rubrics for RLHF/RLVR, and measuring dataset quality and

Qualifications

• 1–4 years hands-on ML data collection and rubric design experience.
• Familiarity with LLM training and evaluation methods.
• Ability to run lightweight experiments and extract insights.
• Comfort working across finance, software and policy domains.
• Practical, builder-oriented approach to data quality.

Responsibilities

• Design data slices across finance, software, and enterprise workflows.
• Build evaluation rubrics and reward signals for RLHF and RLVR pipelines.
• Model annotator behavior and run experiments to improve model capabilities.
• Develop metrics to measure dataset quality, diversity, and impact.
• Create real-world and synthetic data pipelines.
• Collaborate with research teams to translate training objectives into data specifications.

Skills

Data collection design

Evaluation rubric design

LLM training familiarity

Experimentation

Cross-domain collaboration

Job description

True Talent Solutions Partners is seeking a role focused on designing datasets, evaluation rubrics, and reward signals to improve frontier AI models. You will collaborate with research teams to translate training goals into concrete data specs and experiments that inform model training runs.

Responsibilities include crafting data slices across finance, software, and enterprise workflows to reveal failure modes, building assessment rubrics for RLHF/RLVR, and measuring dataset quality and