Remote AI Evaluation Engineer — Code Tasks & Review

Patronus AI; United States; Remote; USD 69,000 - 138,000; Part time

Benefits offered by this job

Flexible schedule

Remote contract

15–20 hours per week

Job summary

Patronus AI is seeking a role-focused engineer to design and assess practical coding and data tasks for AI models. You will craft multi-step exercises from fictional codebases and specifications, then review and score model-generated content against precise rubrics.

Applicants should demonstrate strong coding in multiple languages, technical writing, and the ability to judge work against explicit criteria, while collaborating asynchronously on project tasks.

Qualifications

• Proficient coding ability in one or more major languages (Python/Java/C++/JS/TS/Go).
• Strong ability to review, debug, and improve production code.
• Clear technical writing and ability to evaluate work against criteria.

Responsibilities

• Design original, multi-step tasks using fictional codebases and specs.
• Review generated code for correctness, efficiency, and sound engineering practice.
• Score responses against rubrics and provide structured feedback.
• Refine tasks to reflect real engineering complexity and supervise peers’ work.
• Work independently, meet deadlines, and incorporate feedback.

Skills

Python

Java

C++

JavaScript

TypeScript

Go

Technical writing

Job description

Patronus AI is seeking a role-focused engineer to design and assess practical coding and data tasks for AI models. You will craft multi-step exercises from fictional codebases and specifications, then review and score model-generated content against precise rubrics.

Applicants should demonstrate strong coding in multiple languages, technical writing, and the ability to judge work against explicit criteria, while collaborating asynchronously on project tasks.