Senior Software Engineer, LLM Evaluation (Remote)

AuraOne, Inc.

United States

Remote

USD 83,000 - 165,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

AuraOne, Inc. is seeking a Senior Software Engineer, LLM Evaluation (US-Based, Multi-Language) for remote contract work. The role centers on evaluating production code, reproducing failures, and writing tests to guide model improvements.

You will grade model outputs for correctness and edge cases, document issues, and provide rationale to help the modeling team target gaps. Strong testing and writing skills are essential.

Qualifications

  • Strong day-job engineering experience with reading, running, and debugging unfamiliar code.
  • Comfort writing concise unit tests capturing a single failure mode.
  • Familiarity with a testing framework (pytest, Jest, JUnit, Go test, RSpec, etc.).
  • Clear written reasoning in review notes for senior engineers.

Responsibilities

  • Run and reproduce candidate code outputs in a sandboxed environment.
  • Grade senior LL(M) evaluation solutions for correctness, style, and edge-case handling.
  • Write the smallest failing test demonstrating a model bug.
  • Compare paired solutions with a rubric-based written rationale.
  • Tag failure modes with severity scores and document recurring failures.

Skills

Code review
Debugging
Unit testing
Software engineering judgment
Senior Software Engineer LLM Eval
Python
JavaScript
Java
C++
Rust
Software development

Job description

AuraOne, Inc. is seeking a Senior Software Engineer, LLM Evaluation (US-Based, Multi-Language) for remote contract work. The role centers on evaluating production code, reproducing failures, and writing tests to guide model improvements.

You will grade model outputs for correctness and edge cases, document issues, and provide rationale to help the modeling team target gaps. Strong testing and writing skills are essential.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer, LLM Evaluation & Review (Remote)
Senior Software Engineer, LLM Evaluation & Review (Remote)

AuraOne Human Data • United States

Remote
USD 69,000 - 124,000
Remote work
Contractor engagement
US-eligible
Remote Senior LLM Evaluation & Code-Review Engineer
Remote Senior LLM Evaluation & Code-Review Engineer

AuraOne Human Data • United States

Remote
USD 83,000 - 138,000
Senior LLM Engineer - AI Evaluation (Remote)
Senior LLM Engineer - AI Evaluation (Remote)

AuraOne Human Data • United States

Remote
USD 83,000 - 165,000
Senior Python LLM Evaluation Engineer (Remote)
Senior Python LLM Evaluation Engineer (Remote)

AuraOne Human Data • United States

Remote
USD 96,000 - 179,000
Senior LLM Evaluation Engineer (JS/Python) – Remote
Senior LLM Evaluation Engineer (JS/Python) – Remote

AuraOne Human Data • United States

Remote
USD 83,000 - 165,000
Senior C# Engineer: LLM Evaluation & Code Quality
Senior C# Engineer: LLM Evaluation & Code Quality

AuraOne Human Data • United States

Remote
USD 83,000 - 165,000
Senior Ruby LLM Evaluation & Code-Review Specialist
Senior Ruby LLM Evaluation & Code-Review Specialist

AuraOne Human Data • United States

Remote
USD 110,000 - 165,000
Senior Software Engineer, LLM Evaluation (US-Based, Multi-Language)
Senior Software Engineer, LLM Evaluation (US-Based, Multi-Language)

AuraOne, Inc. • United States

Remote
USD 83,000 - 165,000
Senior Software Engineer, LLM Evaluation (US, Canada, WEU Based)
Senior Software Engineer, LLM Evaluation (US, Canada, WEU Based)

AuraOne Human Data • United States

Remote
USD 69,000 - 124,000
Remote work
Contractor engagement
US-eligible
Remote C/C++ Code Review Engineer for LLM Evaluation
Remote C/C++ Code Review Engineer for LLM Evaluation

AuraOne Human Data • United States

Remote
USD 55,000 - 96,000