Remote AI Benchmark Test Engineer

Mercor; New York (NY); Remote; USD 85,000 - 120,000; Full time

Job summary

Mercor, a leading AI platform, seeks a QA/test engineer to join its GenAI benchmarking effort. You will design checks, review tasks, and debug environments, ensuring edge cases are covered and results are robust.

This is a full-time W-2 role placed with Cincinnatus LLC, fully remote in the United States. You will work about 35 hours per week, collaborating with researchers and task authors to build repeatable quality processes and clear documentation, with independence to tackle ambiguous

Qualifications

• MSc or PhD in STEM or equivalent practical experience.
• 1+ years of experience in test engineering, quality assurance, or a research/software engineering role with strong quality ownership.
• Demonstrated skill designing test cases and quality-review processes, and debugging complex systems end-to-end.
• Working proficiency in Python and Git, and comfort navigating unfamiliar codebases and environments.
• Exceptional attention to detail and clear written documentation habits.
• Past experience in AI training, model evaluation, or quality review of AI-generated work is preferred.
• A perfectionist mindset: creativity in finding what others missed, and the ability to work independently through ambiguous, open-ended problems.
• Ability to engage reliably for approximately 35 hours per week.

Responsibilities

• Design checks: Create test cases that confirm each task works as intended — including the tricky edge cases.
• Review tasks: Give tasks and reference solutions a careful read before they're finalized, catching ambiguity and gaps early.
• Debug: Roll up your sleeves in Python when a task or its checks don't behave the way they should.
• Shape the process: Help build simple, repeatable quality checklists, and share feedback authors can act on right away.
• Protect the results: Watch for shortcuts and grading gaps in AI agent runs so benchmark scores stay trustworthy.

Skills

Test engineering

Quality assurance

Design test cases

Python

Git

Documentation

Independent work

Attention to detail

AI model evaluation

Education

MSc or PhD in STEM

Job description

Mercor, a leading AI platform, seeks a QA/test engineer to join its GenAI benchmarking effort. You will design checks, review tasks, and debug environments, ensuring edge cases are covered and results are robust.

This is a full-time W-2 role placed with Cincinnatus LLC, fully remote in the United States. You will work about 35 hours per week, collaborating with researchers and task authors to build repeatable quality processes and clear documentation, with independence to tackle ambiguous