CPA/Accounting QA Architect for AI Evaluation

Mercor; San Francisco (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Mercor partners with a leading AI research organization to engage experienced accountants. You will define what excellent accounting work looks like by designing task-specific grading criteria and scoring samples with rigorous written justifications.

You will apply consistent, defensible judgment, score both AI-generated and human work, and incorporate feedback from senior reviewers to iterate efficiently.

Qualifications

• Excellent written communication and ability to justify scoring choices clearly.
• Detail-oriented with consistent, defendable judgment under peer review.
• Strong analytical skills applied to real-world accounting work standards.
• Experience with US GAAP, month-end close, reconciliations, and disclosures.

Responsibilities

• Design precise grading criteria for accounting deliverables.
• Score AI-generated and human work with detailed written justifications.
• Apply consistent, evidence-based judgment for reproducibility.
• Incorporate senior reviewers' feedback and iterate quickly.

Skills

Written communication

Attention to detail

Judgment and reasoning

Education

CPA or equivalent certification

Job description

Mercor partners with a leading AI research organization to engage experienced accountants. You will define what excellent accounting work looks like by designing task-specific grading criteria and scoring samples with rigorous written justifications.

You will apply consistent, defensible judgment, score both AI-generated and human work, and incorporate feedback from senior reviewers to iterate efficiently.