Remote AI Benchmark QA Engineer

24-Mag Llc; New York (NY); Remote; USD 76,000 - 117,000; Full time

Job summary

24-MAG LLC is seeking an experienced QA/test engineer to design robust benchmark test cases, review complex tasks, and debug Python environments for frontier AI evaluation work. The role is fully remote within the United States and requires strong attention to detail and independent problem-solving.

Your work will help ensure accurate, reproducible, and defensible evaluation results across multi-step AI tasks, with close collaboration to researchers and task authors.

Qualifications

• At least 1 year of experience in test engineering, quality assurance, software engineering, or related technical roles.
• Experience designing test cases and quality-review processes.
• Strong debugging skills across complex technical systems.
• Working proficiency in Python and Git.
• Ability to work independently on open-ended technical problems.

Responsibilities

• Create comprehensive test cases confirming benchmark tasks function as intended.
• Review complex multi-step tasks and reference solutions for ambiguity and gaps.
• Diagnose failures in Python scripts, test harnesses, and environments.
• Develop repeatable quality processes and checklists for benchmark integrity.
• Collaborate with researchers and task authors to improve evaluation quality.

Skills

Test design

End-to-end debugging

Quality assurance

Attention to detail

Education

Master's degree in STEM

Equivalent practical experience

Tools

Python

Git

Job description

24-MAG LLC is seeking an experienced QA/test engineer to design robust benchmark test cases, review complex tasks, and debug Python environments for frontier AI evaluation work. The role is fully remote within the United States and requires strong attention to detail and independent problem-solving.

Your work will help ensure accurate, reproducible, and defensible evaluation results across multi-step AI tasks, with close collaboration to researchers and task authors.