QA/Test Engineer - Remote AI Benchmarking

mercor; United States; Remote; USD 74,000 - 124,000; Part time

Job summary

Mercor is seeking a QA/Test Engineer for a remote contract role. You will design test cases, review tasks and reference solutions to catch ambiguities, and debug using Python to ensure robust checks.

Build quality checklists and provide actionable feedback to authors, while monitoring AI agent runs for shortcuts and grading gaps to ensure trustworthy benchmark scores. The ideal candidate holds an MSc or PhD in a STEM field (or equivalent experience), has 1+ year in QA/test roles, and

Qualifications

• MSc or PhD in a STEM field or equivalent practical experience.
• 1+ years in test engineering, quality assurance, or a related role.
• Experience designing test cases and debugging complex systems.
• Proficiency in Python and Git
• Exceptional attention to detail and clear documentation habits.

Responsibilities

• Design test cases to confirm task functionality, including edge cases.
• Review tasks and reference solutions to catch ambiguities and gaps early.
• Debug using Python when tasks or checks misbehave.
• Build quality checklists and provide actionable feedback to authors.
• Monitor AI agent runs for shortcuts and grading gaps to ensure trustworthy benchmark scores.

Skills

Python

Attention to detail

Education

MSc or PhD in STEM

Tools

Git

Job description

Mercor is seeking a QA/Test Engineer for a remote contract role. You will design test cases, review tasks and reference solutions to catch ambiguities, and debug using Python to ensure robust checks.

Build quality checklists and provide actionable feedback to authors, while monitoring AI agent runs for shortcuts and grading gaps to ensure trustworthy benchmark scores. The ideal candidate holds an MSc or PhD in a STEM field (or equivalent experience), has 1+ year in QA/test roles, and