Coding & QC Expert

Verita AI; United States; Remote; USD 9,100 - 23,000; Full time

Job summary

Verita AI is seeking a Coding & Rubric QC Expert (Pilot) to audit AI benchmark tasks. You will judge rubric clarity, verifiability, and alignment with the proposed gold standard, marking pass/fail with concise reasoning.

This independent contractor role is remote (North America-based), paid per completed task during a ~1-week pilot with ~10+ hours, and offers potential for continued or expanded work for strong performers. Tasks are paid weekly.

Qualifications

• Based in North America (remote work)
• Bachelor's degree in Computer Science or equivalent
• At least 3 months of human-data/annotation/AI-training work experience
• At least 1 year of software engineering experience

Responsibilities

• Review each rubric claim for clarity, specificity and verifiability.
• Mark each claim as pass, fail, or flagged for follow-up with a short reason.
• Spot missing connections between requirements and questions/answers.
• Double-check that the 'gold' answer is correct before treating it as ground truth.
• Provide an overall accept or reject decision with evidence-based feedback.

Skills

Human-data experience

Software engineering

Education

Bachelor's degree in Computer Science

Job description

Coding & Rubric QC Expert (Pilot)

Engagement type: Independent contractor, paid per completed taskLocation: Remote (must be based in North America)Duration: Paid pilot, ~1 week, with potential for continued or expanded work for strong performersTime commitment: ~10+ hours minimum during the pilot window, flexible scheduling

About the Role

We're looking for detail-oriented reviewers to help evaluate the quality of AI benchmark tasks. Each task gives you a question, a proposed "gold" answer, and a rubric of claims describing what a correct answer should contain. Your job is to judge whether that rubric actually holds up — not to write new content, but to critically audit work that's already been built.

Day to day, this looks like:

• Reviewing each rubric claim and judging whether it's clear, specific, and actually verifiable — or vague, redundant, and unusable.
• Marking each claim as pass, fail, or flagged for follow-up, with a short written reason.
• Spotting requirements that don't actually connect to the question or the answer.
• Double-checking that the "gold" answer is actually correct before treating it as ground truth.
• Giving an overall accept or reject call on the task, backed by your specific findings.

This is evaluative, judgment-driven work rather than open-ended coding or writing. If you enjoy code review, QA, or grading/auditing other people's work, this will feel familiar.

Average task time is expected to be around 20 minutes.

What We're Looking For

Required:

• Based in North America
• Bachelor's degree in Computer Science (or equivalent completed degree)
• At least 3 months of prior experience doing human-data / annotation / AI-training work of any kind
• At least 1 year of software engineering experience outside of human-data work (any language or stack is fine — no specific tech stack required)

Strongly preferred:

• Direct experience grading or auditing rubrics, benchmarks, or "golden set" answers against a reference or acceptance criteria
• Experience critically reviewing someone else's work product — flagging missing requirements, vague criteria, or incorrect reference answers
• A track record of writing clear, evidence-backed feedback rather than simple pass/fail calls with no explanation
• Prior experience as a task reviewer or QC lead on another data-annotation project
• Prior experience specifically on coding-related annotation projects

Compensation & Pay

• Paid per completed, accepted task (not hourly)

Verita engagements are remote and asynchronous, and Verita says they are paid weekly.