Coding & QC Expert

Remolity, Inc.; United States; Remote; USD 21,000 - 34,000; Part time

Benefits offered by this job

Paid pilot

Flexible scheduling

Remote work within eligible countries

Job summary

Verita AI is seeking Coding & QC Experts for a paid pilot evaluating the quality of AI benchmark tasks. Each task includes a question, a proposed gold answer and a rubric of claims describing what a correct answer should contain.

Critically audit existing work, assess rubric claims, and verify the reference answer before treating it as ground truth. This evaluative work suits reviewers who enjoy code review, QA, and auditing others' work.

Qualifications

• Bachelor's degree in Computer Science or equivalent.
• At least 3 months of human-data, annotation or AI-training experience.
• At least 1 year of software engineering experience outside human-data work.
• Based in United States or Canada.

Responsibilities

• Review every rubric claim for clarity, specificity and verifiability.
• Mark claims as pass, fail or flagged for follow-up with a short written reason.
• Identify requirements that do not connect to the question or proposed answer.
• Check the correctness of the proposed gold answer before treating it as ground truth.
• Provide an overall accept or reject decision for each task, supported by findings.

Skills

Code review

Quality assurance

Auditing

Education

Bachelor's degree in Computer Science

Job description

Coding & QC Expert

Remote — United States, Canada only

Code review Rubric evaluation Quality control AI benchmarks Software engineering Data annotation Evidence-based feedback

Special Referral gives your application an enhanced referral route, helping it stand out and potentially improving your chances of being considered.

About the role

Verita AI is seeking Coding & Rubric QC Experts for a paid pilot evaluating the quality of AI benchmark tasks. Each task includes a question, a proposed gold answer and a rubric of claims describing what a correct answer should contain.

Critically audit existing work rather than create new content. Assess whether rubric claims hold up and whether the reference answer is actually correct.

This is evaluative, judgment-driven work suited to reviewers who enjoy code review, quality assurance and auditing other people's work. Strong performers may be considered for continued or expanded work after the approximately one-week pilot.

Scope of Work

• Review every rubric claim for clarity, specificity and verifiability, identifying vague, redundant or unusable criteria.
• Mark claims as pass, fail or flagged for follow-up, with a short written reason.
• Identify requirements that do not connect to the question or proposed answer.
• Check the correctness of the proposed gold answer before treating it as ground truth.
• Give an overall accept or reject decision on each task, supported by specific findings.

What you’ll bring

• Be based in the United States or Canada under the combined role-specific North America requirement and Verita AI partner policy.
• Completed bachelor's degree in Computer Science, or an equivalent completed degree.
• At least 3 months of human-data, annotation or AI-training experience of any kind.
• At least 1 year of software engineering experience outside human-data work. Any language or stack is acceptable; no specific stack is required.

Preferred Qualifications

• Direct experience grading or auditing rubrics, benchmarks or golden-set answers against references or acceptance criteria.
• Experience critically reviewing others' work and flagging missing requirements, vague criteria or incorrect reference answers.
• A record of clear, evidence-backed feedback rather than unexplained pass/fail decisions.
• Experience as a task reviewer or QC lead on a data-annotation project.
• Experience on coding-related annotation projects.
• Remote work within eligible countries

Benefits

• Paid pilot
• Flexible scheduling
• Remote work within eligible countries

Schedule and availability

Approximately 10+ hours minimum during a paid pilot of about one week, with flexible scheduling. Average task time is expected to be around 20 minutes.

Contract & Payment Terms

• Independent contractor engagement paid per completed, accepted task, not hourly.
• Payment is per completed, accepted task, not for total hours worked.
• Paid pilot of approximately one week, with potential for continued or expanded work for strong performers.

Where you can work

Verita AI only accepts applications from candidates based in the United States (USA), United Kingdom (UK) and Canada. This restriction applies to all current and future Verita AI opportunities. This role additionally requires North America residence. Combined with Verita AI's partner-wide country policy, only USA and Canada applicants are eligible for this opportunity; UK applicants are not eligible for this role.

This role is open to people based in: United States, Canada.

Remote does not always mean work from any country. Always check the employer’s location and working-hour requirements.