QA Engineer - AI Benchmarking (Remote, 35h/wk)

Weekday AI; United States; Remote; USD 83,000 - 124,000; Full time

Job summary

Weekday AI is seeking an experienced QA and Test Engineer for a fully remote, independent contractor role in the United States. The position focuses on building reliable, reproducible AI evaluation benchmarks and validating complex evaluation tasks with strong debugging of Python-based scripts.

You will design test cases, review tasks for ambiguity, and collaborate with AI researchers to ensure benchmark integrity and rigorous testing methodologies. approx.

Qualifications

• Master's degree, PhD, or equivalent practical experience in a STEM discipline involving software engineering, research, or advanced technical problem solving.
• Minimum 1 year of professional experience in Quality Assurance, Test Engineering, Software Engineering, Research Engineering, or related field with strong quality ownership.
• Proven experience designing test cases, validating complex software systems, and debugging end-to-end workflows.
• Strong proficiency in Python and Git, with the ability to troubleshoot unfamiliar codebases and technical environments.
• Excellent analytical thinking, problem-solving skills, and exceptional attention to detail.
• Experience documenting bugs, test strategies, and technical findings with clear written communication.
• Experience evaluating AI systems, machine learning models, or AI-generated outputs is preferred.
• Ability to work independently while managing multiple complex tasks with minimal supervision.
• Ability to commit approximately 35 hours per week on a consistent basis.

Responsibilities

• Design comprehensive test cases that validate evaluation tasks, including complex edge cases and unexpected scenarios.
• Review benchmark tasks and reference solutions to identify ambiguity, inconsistencies, missing requirements, and grading gaps.
• Debug task environments and Python-based evaluation scripts to ensure reliable execution and accurate results.
• Develop repeatable quality assurance processes, validation checklists, and testing frameworks for benchmark creation.
• Identify potential shortcuts, exploits, or weaknesses that could compromise evaluation accuracy or benchmark integrity.
• Collaborate with AI researchers, engineers, and task authors to improve task quality, reproducibility, and technical rigor.

Skills

Python

Git

Test design

Edge case analysis

Education

Master's degree or PhD in STEM

Job description

Weekday AI is seeking an experienced QA and Test Engineer for a fully remote, independent contractor role in the United States. The position focuses on building reliable, reproducible AI evaluation benchmarks and validating complex evaluation tasks with strong debugging of Python-based scripts.

You will design test cases, review tasks for ambiguity, and collaborate with AI researchers to ensure benchmark integrity and rigorous testing methodologies. approx.