Remote AI Code Trace Evaluator (Contract)

OpenTrain AI; Northern (KY); Hybrid; USD 96,000 - 124,000; Part time

Job summary

OpenTrain AI is seeking an AI Software Development Trace Evaluator to review AI-assisted coding sessions used in model training and evaluation. You will assess complete development workflows, including code correctness, engineering practices, and the quality of reasoning shown throughout each trajectory.

Your written, rubric-based judgments will support the evaluation of software-development traces for advanced AI systems.

Qualifications

• At least three years of professional software development experience.
• Hands-on experience with AI-assisted coding tools such as Cursor, GitHub Copilot, Claude Code, or similar products.
• Strong code-reading and debugging ability across backend or full-stack software systems.
• Ability to evaluate coding trajectories for correctness, sound workflows, and engineering best practices.
• Ability to write clear, detailed, rubric-based technical feedback.
• Familiarity with agentic or specification-driven development workflows.
• Availability for at least 20 hours per week.

Responsibilities

• Review end-to-end coding sessions produced with AI-assisted developer tools.
• Evaluate multi-step coding trajectories for correctness and workflow soundness.
• Assess reasoning quality and adherence to engineering best practices.
• Read and debug code across backend or full-stack systems.
• Identify defects and assess implementation quality.
• Write clear, detailed feedback that explains evaluation decisions.
• Apply rubrics consistently across software-development traces.
• Examine AI-assisted tools in agentic or specification-driven development workflows.

Skills

Software development

AI-assisted coding tools

Code reading

Debugging

Technical feedback

Agentic development workflows

Availability

Tools

Cursor

GitHub Copilot

Claude Code

Job description

OpenTrain AI is seeking an AI Software Development Trace Evaluator to review AI-assisted coding sessions used in model training and evaluation. You will assess complete development workflows, including code correctness, engineering practices, and the quality of reasoning shown throughout each trajectory.

Your written, rubric-based judgments will support the evaluation of software-development traces for advanced AI systems.