DevDocs QA Specialist — AI-Generated API & CLI Docs

Mercor; United States; Remote; USD 55,000 - 110,000; Part time

Job summary

Mercor is seeking a detail-oriented evaluator to review AI-generated developer-facing writing (CLI help text, error messages, API references) against a client task brief and writing guide. You will read diffs and terminal output to spot incorrect explanations and select a preferred sample.

You will document a one-line reason with tags and complete a calibration gate using a known-answer test set before moving to live data.

Qualifications

• Experience evaluating AI-generated writing, including CLI help text, error messages, or API references.
• Ability to read task briefs and guides and identify technical inaccuracies.

Responsibilities

• Review AI-generated developer-facing text against briefs and logs to catch wrong explanations.
• Select preferred sample or flag ties; document a one-line justification with tags.
• Run calibration against known-answer test sets before live data.

Job description

Mercor is seeking a detail-oriented evaluator to review AI-generated developer-facing writing (CLI help text, error messages, API references) against a client task brief and writing guide. You will read diffs and terminal output to spot incorrect explanations and select a preferred sample.

You will document a one-line reason with tags and complete a calibration gate using a known-answer test set before moving to live data.