Expert: Software Engineer – AI Evaluation (Task-based Contract, Remote)

Patronus AI; United States; Remote; USD 69,000 - 138,000; Part time

Benefits offered by this job

Flexible schedule

Remote contract

15–20 hours per week

Job summary

Patronus AI is seeking a role-focused engineer to design and assess practical coding and data tasks for AI models. You will craft multi-step exercises from fictional codebases and specifications, then review and score model-generated content against precise rubrics.

Applicants should demonstrate strong coding in multiple languages, technical writing, and the ability to judge work against explicit criteria, while collaborating asynchronously on project tasks.

Qualifications

• Proficient coding ability in one or more major languages (Python/Java/C++/JS/TS/Go).
• Strong ability to review, debug, and improve production code.
• Clear technical writing and ability to evaluate work against criteria.

Responsibilities

• Design original, multi-step tasks using fictional codebases and specs.
• Review generated code for correctness, efficiency, and sound engineering practice.
• Score responses against rubrics and provide structured feedback.
• Refine tasks to reflect real engineering complexity and supervise peers’ work.
• Work independently, meet deadlines, and incorporate feedback.

Skills

Python

Java

C++

JavaScript

TypeScript

Go

Technical writing

Job description

Role overview

Create and assess practical engineering exercises that reveal how well AI models handle coding, technical reasoning, and detailed instructions. You’ll draw on professional software development judgment to build realistic scenarios, define what good work looks like, and explain where model responses succeed or fall short.

Responsibilities

• Design original, multi-step tasks using fictional codebases, technical specifications, architecture documents, or datasets.
• Review generated code and technical content for correctness, efficiency, clarity, and sound engineering practice.
• Score responses against reference answers and rubrics, providing specific, structured feedback.
• Refine tasks to reflect real engineering complexity and review other contributors’ work for accuracy and practical relevance.
• Work independently and asynchronously, meet deadlines, and incorporate reviewer feedback.

Requirements

• At least two years of professional software engineering experience; internships do not count.
• Strong coding ability in one or more major languages, such as Python, Java, C++, JavaScript, TypeScript, or Go.
• Experience with production software, including building, reviewing, or debugging code.
• Familiarity with system design, data pipelines, machine learning, or applications built with large language models.
• Clear technical writing and the ability to evaluate work consistently against explicit criteria.

Nice to have

• Background in code review, technical writing, AI training, or model evaluation.
• Experience creating realistic engineering tasks and assessing their difficulty.

Benefits and work setup

• Remote, task-based contract open to applicants in India; this is not a full-time position.
• Flexible schedule, typically around 15–20 hours per week, with work that may fit alongside a full-time role.
• Pay is listed at $50–$100 per hour, with some tasks paid at a flat rate.
• Work must be original and human-authored. Use fictional scenarios and do not include employer or client materials, NDA-protected content, or copied published work.