Senior LLM Engineer - AI Evaluation (Remote)

AuraOne Human Data

United States

Remote

USD 83,000 - 165,000

Part time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

AuraOne, Inc. is seeking a Senior LLM Engineer - AI Evaluation to join a remote engineering review track. You will reproduce failures, create minimal failing tests, and explain fixes so the modeling team can target gaps.

Ideal candidates read, run, and debug unfamiliar code, write concise unit tests, and provide clear, evidence-based reviews. This is a CONTRACTOR role with US eligibility, paid hourly after interview.

Qualifications

  • Proven ability to read, run, and debug unfamiliar code.
  • Experience with a testing framework (pytest, Jest, JUnit, etc.).
  • Excellent written reasoning to justify reviews.

Responsibilities

  • Run and reproduce candidate code outputs in a sandboxed environment.
  • Grade senior llm engineer solutions for correctness, style, and edge-case handling.
  • Write the smallest failing test that demonstrates a model’s bug.
  • Compare paired solutions and rank them with a rubric-based rationale.
  • Tag failure modes (compile error, runtime crash, edge-case) with severity scores.
  • Document recurring code-generation failures to guide improvements.
  • Calibrate against gold-standard reviews to maintain inter-rater reliability.

Skills

Code review
Debugging
Unit testing
Engineering judgment
Machine Learning
Artificial Intelligence
Software development

Job description

AuraOne, Inc. is seeking a Senior LLM Engineer - AI Evaluation to join a remote engineering review track. You will reproduce failures, create minimal failing tests, and explain fixes so the modeling team can target gaps.

Ideal candidates read, run, and debug unfamiliar code, write concise unit tests, and provide clear, evidence-based reviews. This is a CONTRACTOR role with US eligibility, paid hourly after interview.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, LLM Evaluation (Remote)
Senior Software Engineer, LLM Evaluation (Remote)

AuraOne, Inc. • United States

Remote
USD 83,000 - 165,000
Senior Software Engineer, LLM Evaluation & Review (Remote)
Senior Software Engineer, LLM Evaluation & Review (Remote)

AuraOne Human Data • United States

Remote
USD 69,000 - 124,000
Remote work
Contractor engagement
US-eligible
Senior LLM Engineer - AI Evaluation
Senior LLM Engineer - AI Evaluation

AuraOne Human Data • United States

Remote
USD 83,000 - 165,000
Remote Senior LLM Evaluation & Code-Review Engineer
Remote Senior LLM Evaluation & Code-Review Engineer

AuraOne Human Data • United States

Remote
USD 83,000 - 138,000
Remote AI Evaluation Engineer — Civil Code Reviewer
Remote AI Evaluation Engineer — Civil Code Reviewer

AuraOne Human Data • United States

Remote
USD 102,000 - 146,000
Senior C# Engineer: LLM Evaluation & Code Quality
Senior C# Engineer: LLM Evaluation & Code Quality

AuraOne Human Data • United States

Remote
USD 83,000 - 165,000
Remote ML Engineer - LLM & Agentic Evaluations (Contractor)
Remote ML Engineer - LLM & Agentic Evaluations (Contractor)

AuraOne Human Data • United States

Remote
USD 83,000 - 165,000
Remote Senior AI Data Review Engineer
Remote Senior AI Data Review Engineer

AuraOne, Inc. • United States

Remote
USD 141,000 - 190,000
Senior Software Engineer, AI Evaluation & Coding Agents
Senior Software Engineer, AI Evaluation & Coding Agents

AuraOne Human Data • United States

Remote
USD 138,000 - 207,000
Remote Electrical Engineer Reviewer — AI Code Evaluation
Remote Electrical Engineer Reviewer — AI Code Evaluation

AuraOne Human Data • United States

Remote
USD 106,000 - 143,000