Software Engineer, RL Environments

True Talent Solutions Partners; San Francisco (CA); On-site; USD 260,000 - 290,000; Full time

Job summary

True Talent Solutions Partners is seeking a role focused on designing datasets, evaluation rubrics, and reward signals to improve frontier AI models. You will collaborate with research teams to translate training goals into concrete data specs and experiments that inform model training runs.

Responsibilities include crafting data slices across finance, software, and enterprise workflows to reveal failure modes, building assessment rubrics for RLHF/RLVR, and measuring dataset quality and

Qualifications

• 1–4 years hands-on ML data collection and rubric design experience.
• Familiarity with LLM training and evaluation methods.
• Ability to run lightweight experiments and extract insights.
• Comfort working across finance, software and policy domains.
• Practical, builder-oriented approach to data quality.

Responsibilities

• Design data slices across finance, software, and enterprise workflows.
• Build evaluation rubrics and reward signals for RLHF and RLVR pipelines.
• Model annotator behavior and run experiments to improve model capabilities.
• Develop metrics to measure dataset quality, diversity, and impact.
• Create real-world and synthetic data pipelines.
• Collaborate with research teams to translate training objectives into data specifications.

Skills

Data collection design

Evaluation rubric design

LLM training familiarity

Experimentation

Cross-domain collaboration

Job description

About the Role

Design datasets, evaluation rubrics, and reward signals that help improve frontier AI models. You will work with research teams to turn training goals into practical experiments and data specifications, with results feeding into model training runs.

What You’ll Do

• Design data slices across finance, software, and enterprise workflows to reveal meaningful model failure modes.

• Build and refine evaluation rubrics and reward signals for RLHF and RLVR pipelines.

• Model annotator behavior and run experiments to improve model capabilities.

• Develop quantitative approaches to measure dataset quality, diversity, and downstream impact.

• Create and manage real-world and synthetic data pipelines.

• Partner with research teams to translate training objectives into data and evaluation specifications.

What We’re Looking For

• One to four years of hands-on experience designing machine learning data collection strategies and evaluation rubrics.

• Strong familiarity with LLM training and evaluation methods, including experience creating data slices to surface model failure modes.

• Ability to run lightweight experiments, work through ambiguous results, and extract actionable insights.

• Comfort working across domains such as finance, software engineering, and policy.

• A practical, builder-oriented approach and a strong interest in how data structure and quality shape model behavior.

Compensation & Benefits

Base salary: $260,000 to $290,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California.