Frontier AI Evaluation Engineer (RL & Data Pipelines)

Prime Intellect, Inc.; San Francisco (CA); Hybrid; USD 150,000 - 350,000; Full time

Benefits offered by this job

Cash compensation range

Remote or on-site SF office

Visa sponsorship

Off-sites and learning opportunities

Job summary

Prime Intellect, Inc. is seeking a hands-on engineer to design and build evaluations for coding tasks and long-horizon agents, working directly with researchers to diagnose failures and improve task design.

You will scale RL data pipelines, develop verifiers, and implement scoring logic, tracing model outputs to distinguish weaknesses in data, environment, or scoring, in a highly collaborative setting. Flexibility to work remotely or from our San Francisco offices, visa sponsorship for

Qualifications

• Strong software engineering skills in Python and experience building data pipelines, evaluation systems, or ML workflows.
• Hands-on experience with language models or agents and understanding how data/evaluation affect model behavior.
• Ability to inspect complex data and model outputs, identify failure modes, and design testable measurements.
• Proven track record owning technical projects from experimentation to validation in collaboration with researchers/engineers.
• Experience building technical evaluations or agent environments with inspectable traces and scoring code.

Responsibilities

• Design and Build evaluations for coding tasks and long-horizon agents.
• Build executable environments in Verifiers with realistic task setups and clear success criteria.
• Develop and test scoring logic and verifiers, investigating reward design and misleading progress measures.
• Trace failures through model outputs and harness code to distinguish model limitations from environment/scoring issues.
• Build and scale RL data pipelines that generate and curate challenging tasks for agents.
• Collaborate with RL researchers to test task/training data changes on held-out tasks.

Skills

Python

Data pipelines

ML evaluation systems

Research collaboration

Debugging model outputs

Tools

Python

Verifiers

RL pipelines

Job description

Prime Intellect, Inc. is seeking a hands-on engineer to design and build evaluations for coding tasks and long-horizon agents, working directly with researchers to diagnose failures and improve task design.

You will scale RL data pipelines, develop verifiers, and implement scoring logic, tracing model outputs to distinguish weaknesses in data, environment, or scoring, in a highly collaborative setting. Flexibility to work remotely or from our San Francisco offices, visa sponsorship for