Frontier AI Evaluation Engineer

OpenKrill; United States; Hybrid; USD 150,000 - 350,000; Full time

Benefits offered by this job

Remote or office options

Visa sponsorship

Relocation assistance

Team off-sites

Competitive compensation

Job summary

Prime Intellect is building open frontier AI with a full-stack platform for post-training at frontier scale. You will design evaluations for coding and long-horizon agents, and build executable environments with clear success criteria in collaboration with researchers.

Join a team that scales RL data pipelines, tests task changes for improved performance on held-out tasks, and troubleshoots model traces to distinguish environment issues from model limitations.

Qualifications

• Strong software engineering skills in Python and data pipelines.
• Experience with ML evaluation workflows and agents.
• Ability to inspect model outputs and diagnose failures.
• Track record of owning technical projects with researchers.
• Experience building evaluations or agent environments.

Responsibilities

• Design and build challenging evaluations for coding and long-horizon agents.
• Build executable environments in Verifiers with clear success criteria.
• Develop and test scoring logic and verifiers, investigating reward hacks.
• Trace failures through model outputs and harness code.
• Scale RL data pipelines generating coding and agentic tasks.
• Collaborate with RL researchers to test task & data changes.

Skills

Python programming

Data pipelines

ML evaluation

Language models/agents

Data analysis

Tools

Verifiers

Job description

Prime Intellect is building open frontier AI with a full-stack platform for post-training at frontier scale. You will design evaluations for coding and long-horizon agents, and build executable environments with clear success criteria in collaboration with researchers.

Join a team that scales RL data pipelines, tests task changes for improved performance on held-out tasks, and troubleshoots model traces to distinguish environment issues from model limitations.