Remote SWE Agent Evaluation Engineer - Code Review

AuraOne; United States; On-site; USD 55,104 - 96,432; Full time

Benefits offered by this job

Remote work

Job summary

AuraOne is seeking a SWE Agent Evaluation Specialist for a remote, US-eligible contractor role. You will review produced code, reproduce results in a sandbox, and judge correctness, complexity, and edge-case handling.

You’ll write minimal failing tests to demonstrate model gaps and compare paired solutions with a rationale aligned to a rubric. You should be able to read and debug unfamiliar code, perform concise unit tests, and communicate clear engineering reasoning.

Qualifications

• Strong day-job engineering experience; able to read, run, and debug unfamiliar code for SWE Agent Evaluation Specialist work.
• Comfortable writing concise unit tests that capture a single failure mode.
• Familiarity with at least one: pytest, Jest, JUnit, Go test, RSpec, or equivalent.

Responsibilities

• Run and reproduce candidate code outputs in a sandboxed environment for SWE Agent Evaluation Specialist assignments.
• Grade SWE Agent Evaluation engineering solutions for correctness, complexity, idiomatic style, and edge-case handling.
• Write minimal failing tests that demonstrate the bug a model output missed.
• Compare paired solutions and rank them with a written rationale tied to the rubric.
• Tag failure modes (compile error, runtime crash, off-by-one, security issue) with severity scores.
• Document recurring code-generation failures so the modeling team can target them.
• Calibrate against gold-standard reviews to keep inter-rater agreement above target.

Skills

Code review

Debugging

Unit testing

Software engineering judgment

SWE Agent Evaluation engineering

Software testing

Agent evaluation

Agent

Tools

pytest

Jest

JUnit

Go test

RSpec

Job description

AuraOne is seeking a SWE Agent Evaluation Specialist for a remote, US-eligible contractor role. You will review produced code, reproduce results in a sandbox, and judge correctness, complexity, and edge-case handling.

You’ll write minimal failing tests to demonstrate model gaps and compare paired solutions with a rationale aligned to a rubric. You should be able to read and debug unfamiliar code, perform concise unit tests, and communicate clear engineering reasoning.