Remote QA/Test Engineer — AI Benchmark Validation

Weekday AI (YC W21); United States; On-site; USD 83,000 - 124,000; Full time

Job summary

Weekday AI (YC W21) in the United States seeks an experienced QA/Test Engineer to establish quality standards and testing processes for agentic evaluation benchmarks in advanced AI models. You will design tests, review task quality, debug environments with Python, and collaborate with researchers.

This fully remote role requires approximately 35 hours per week and is offered as Full-Time employment. You will help ensure reliable, unambiguous benchmarking and rigorous evaluation of AI tasks.

Qualifications

• MSc or PhD in STEM or equivalent research/engineering experience.
• 1+ year in QA or test engineering with ownership of quality.
• Strong ability to design test cases, QA processes, and end-to-end investigation.

Responsibilities

• Design test cases for benchmark tasks, including edge cases and failures.
• Review task quality, requirements, and reference solutions for ambiguities.
• Debug and troubleshoot environments using Python and development tools.
• Develop repeatable QA processes and checklists.
• Identify evaluation gaps and ensure reliable scoring.
• Collaborate with researchers and task authors to validate fixes.
• Maintain high-quality standards across evaluation datasets.

Skills

Python

Git

Test design

Debugging

Documentation

Independent work

Attention to detail

Education

MSc or PhD in STEM

Job description

Weekday AI (YC W21) in the United States seeks an experienced QA/Test Engineer to establish quality standards and testing processes for agentic evaluation benchmarks in advanced AI models. You will design tests, review task quality, debug environments with Python, and collaborate with researchers.

This fully remote role requires approximately 35 hours per week and is offered as Full-Time employment. You will help ensure reliable, unambiguous benchmarking and rigorous evaluation of AI tasks.