Remote SRE AI Evaluator: Code Review & Debugging

AuraOne; United States; On-site; USD 83,000 - 131,000; Full time

Job summary

AuraOne is seeking a Site Reliability Engineering AI Evaluator for a remote contractor role focused on evaluating production code, debugging traces, and AI outputs against correctness standards.

You will reproduce failures, write minimal unit tests for edge cases, and document review notes to guide improvements. Strong debugging, code review, and testing skills are essential; familiarity with pytest/Jest/JUnit is a plus. Remote US-eligible; compensation hourly after interview.

Qualifications

• Strong day-to-day engineering experience with reading, running, and debugging unfamiliar code.
• Ability to write concise unit tests that capture a single failure mode.
• Familiarity with pytest, Jest, JUnit, Go test, RSpec, or equivalent.
• Clear written reasoning to convince senior engineers.
• Reliable async availability for at least 10 hours per week.
• Prior code-review or technical-interview grading experience is a plus.

Responsibilities

• Run and reproduce candidate code outputs in a sandboxed environment for Site Reliability Engineering AI Evaluator assignments.
• Grade cloud & platform engineering solutions for correctness, complexity, idiomatic style, and edge-case handling.
• Write minimal failing tests that demonstrate the bug a model output missed.
• Compare paired solutions and rank them with a written rationale tied to the rubric.
• Tag failure modes (compile error, runtime crash, off-by-one, security issue) with severity scores.
• Document recurring code-generation failures so the modeling team can target them.
• Calibrate against gold-standard reviews to keep inter-rater agreement above target.

Skills

Code review

Debugging

Unit testing

Engineering judgment

Cloud & platform engineering

Software engineering

AI evaluation

Rubric writing

Expert review

Job description

AuraOne is seeking a Site Reliability Engineering AI Evaluator for a remote contractor role focused on evaluating production code, debugging traces, and AI outputs against correctness standards.

You will reproduce failures, write minimal unit tests for edge cases, and document review notes to guide improvements. Strong debugging, code review, and testing skills are essential; familiarity with pytest/Jest/JUnit is a plus. Remote US-eligible; compensation hourly after interview.