Remote AI Benchmark QA Engineer

Weekday 1; United States; Remote; USD 83,000 - 124,000; Full time

Job summary

Weekday 1 is seeking an experienced QA and Test Engineer to join a pioneering AI initiative that builds next-generation evaluation benchmarks for frontier AI models. You will review complex, multi-step tasks, validate correctness, identify edge cases, and strengthen testing before benchmarks are deployed.

This fully remote, full-time role requires about 35 hours per week. You will work closely with AI researchers and task authors to improve evaluation quality, reduce ambiguity, and ensure

Qualifications

• Master’s degree, PhD, or equivalent in STEM with software engineering, research or advanced technical problem solving.
• Minimum 1 year of professional experience in QA, Test Eng., Software Eng., Research Eng., or related field with strong ownership.
• Proven experience designing test cases, validating complex software systems, and debugging end-to-end workflows.
• Strong proficiency in Python and Git, with ability to troubleshoot unfamiliar codebases.
• Excellent analytical thinking, problem-solving and attention to detail.
• Experience documenting bugs, test strategies, and findings with clear written communication.
• Experience evaluating AI systems, ML models, or AI outputs is preferred.
• Ability to work independently while managing multiple tasks and ~35 hrs/week.

Responsibilities

• Design comprehensive test cases validating evaluation tasks, including edge cases.
• Review benchmark tasks and reference solutions to identify ambiguity or gaps.
• Debug task environments and Python-based evaluation scripts for reliable results.
• Develop repeatable QA processes, checklists, and testing frameworks for benchmark creation.
• Identify shortcuts, exploits, or weaknesses impacting evaluation accuracy.
• Collaborate with researchers, engineers, and task authors to improve task quality and reproducibility.

Skills

Python

Git

Analytical thinking

Attention to detail

Independent work

Education

Master's/PhD or equivalent STEM

Job description

Weekday 1 is seeking an experienced QA and Test Engineer to join a pioneering AI initiative that builds next-generation evaluation benchmarks for frontier AI models. You will review complex, multi-step tasks, validate correctness, identify edge cases, and strengthen testing before benchmarks are deployed.

This fully remote, full-time role requires about 35 hours per week. You will work closely with AI researchers and task authors to improve evaluation quality, reduce ambiguity, and ensure