AI Software Development Trace Evaluator

OpenTrain AI; Northern (KY); Hybrid; USD 96,000 - 124,000; Part time

Job summary

OpenTrain AI is seeking an AI Software Development Trace Evaluator to review AI-assisted coding sessions used in model training and evaluation. You will assess complete development workflows, including code correctness, engineering practices, and the quality of reasoning shown throughout each trajectory.

Your written, rubric-based judgments will support the evaluation of software-development traces for advanced AI systems.

Qualifications

• At least three years of professional software development experience.
• Hands-on experience with AI-assisted coding tools such as Cursor, GitHub Copilot, Claude Code, or similar products.
• Strong code-reading and debugging ability across backend or full-stack software systems.
• Ability to evaluate coding trajectories for correctness, sound workflows, and engineering best practices.
• Ability to write clear, detailed, rubric-based technical feedback.
• Familiarity with agentic or specification-driven development workflows.
• Availability for at least 20 hours per week.

Responsibilities

• Review end-to-end coding sessions produced with AI-assisted developer tools.
• Evaluate multi-step coding trajectories for correctness and workflow soundness.
• Assess reasoning quality and adherence to engineering best practices.
• Read and debug code across backend or full-stack systems.
• Identify defects and assess implementation quality.
• Write clear, detailed feedback that explains evaluation decisions.
• Apply rubrics consistently across software-development traces.
• Examine AI-assisted tools in agentic or specification-driven development workflows.

Skills

Software development

AI-assisted coding tools

Code reading

Debugging

Technical feedback

Agentic development workflows

Availability

Tools

Cursor

GitHub Copilot

Claude Code

Job description

About OpenTrain

OpenTrain is the #1 platform for finding and building careers in AI training and data labeling. OpenTrain AI hires and contracts contributors for specialized projects where human expertise helps improve advanced artificial intelligence systems.

As an AI Software Development Trace Evaluator, you will use your professional engineering experience to assess how AI-assisted developer tools handle real software-development workflows. Your evaluations will help make AI systems more capable, reliable, and useful for developers.

• Remote contract work with OpenTrain AI
• Candidates must be located in the United States
• Pay ranges from $70 to $90 per hour
• Part-time contract classification with an expected commitment of about 40 hours per week

About AI Training and Software Evaluation

AI training is the human side of building artificial intelligence. People review examples, assess model outputs, and provide structured feedback so AI systems can learn to produce more accurate and useful results.

In software-focused AI training, contributors examine generated code and complete development trajectories rather than evaluating isolated answers. This work gives experienced developers a direct role in shaping how advanced AI systems reason about implementation, debugging, and engineering practice.

• Review AI-generated software-development work
• Assess reasoning and decisions across complete coding workflows
• Apply consistent rubrics to support model evaluation
• Work remotely with flexible AI training opportunities

The Role

OpenTrain is recruiting an AI Software Development Trace Evaluator to review AI-assisted coding sessions used in model training and evaluation. You will assess complete development workflows, including code correctness, engineering practices, and the quality of reasoning shown throughout each trajectory.

Your written, rubric-based judgments will support the evaluation of software-development traces for advanced AI systems. The work requires strong technical judgment, careful code analysis, and the ability to explain decisions clearly.

• Role type: Remote contractor
• Location: United States
• Expected availability: 20+ hours per week, with about 40 hours per week expected for this role
• Pay: $70-$90 per hour
• Language: English

What You'll Do

You will review end-to-end coding sessions produced with AI-assisted developer tools. Your assessments will cover both the resulting software and the process used to create it, including multi-step decisions, tool usage, and adherence to sound development practices.

• Review complete coding sessions produced with AI-assisted developer tools.
• Evaluate multi-step coding trajectories for correctness and workflow soundness.
• Assess reasoning quality and adherence to engineering best practices.
• Read and debug code across backend or full-stack software systems.
• Identify defects and assess implementation quality.
• Write clear, detailed feedback that explains evaluation decisions.
• Apply rubrics consistently across software-development traces.
• Examine AI-assisted tools in agentic or specification-driven development workflows.

Required Qualifications

This role requires at least three years of professional software development experience, despite the associated entry-level experience classification. You should be comfortable reading and debugging code across backend or full-stack systems and judging whether multi-step coding work is correct, maintainable, and aligned with accepted engineering practices.

• At least three years of professional software development experience.
• Hands-on experience with AI-assisted coding tools such as Cursor, GitHub Copilot, Claude Code, or similar products.
• Strong code-reading and debugging ability across backend or full-stack software systems.
• Ability to evaluate coding trajectories for correctness, sound workflows, and engineering best practices.
• Ability to write clear, detailed, rubric-based technical feedback.
• Familiarity with agentic or specification-driven development workflows.
• Availability for at least 20 hours per week.

Helpful Background

The following experience is helpful but is not listed as required. It may help you evaluate development traces involving specialized tools, prior assessment processes, or developer-focused workflows.

• Experience with Kiro or Amazon CodeCatalyst.
• Previous evaluation or grading of AI-generated code.
• Contributions to developer tooling.

Build Your AI Training Career With OpenTrain

AI training and data-labeling work is a fast-growing way to apply technical expertise to cutting-edge technology. OpenTrain helps contributors discover projects, build a profile, and develop a durable portfolio of AI training experience.

Creating an OpenTrain account is free. By completing projects like this one, you can demonstrate credible software evaluation experience and grow your career in the human side of AI development.