Evaluation QA Lead (Agentic and RAG Evaluation)

Vettedlance; United States; Remote; USD 83,000 - 124,000; Part time

Job summary

Vettedlance is seeking an experienced QA lead to run quality for AI evaluation projects, including rubric design, reviewer calibration and tracking agreement for agentic and RAG systems.

You will own QA for agentic-workflow, write and refine rubrics and gold sets, calibrate reviewers and monitor inter-annotator agreement, audit samples, and report quality trends. Remote, hourly contract, part-time, 10-40 hrs/week.

Qualifications

• 4+ years owning QA on annotation, evaluation or software testing.
• Hands-on experience evaluating LLM, agent or RAG systems.
• Comfortable with agreement metrics and spreadsheets.
• Fluent written English.

Responsibilities

• Own QA for agentic-workflow and RAG/conversational evaluation projects.
• Write and refine rubrics and gold sets.
• Calibrate reviewers and track inter-annotator agreement (kappa, alpha).
• Audit samples and report quality trends.

Skills

QA leadership

Rubric design

Reviewer calibration

Inter-annotator agreement

Agreement tracking

LLM evaluation

RAG systems

Job description

Experienced QA leads run quality for AI evaluation projects: rubric design, reviewer calibration and agreement tracking for agentic and RAG systems.

What you will do

• Own QA for agentic-workflow and RAG/conversational evaluation projects.
• Write and refine rubrics and gold sets.
• Calibrate reviewers and track inter-annotator agreement (kappa, alpha).
• Audit samples and report quality trends.

What you need

• 4+ years owning QA on annotation, evaluation or software testing.
• Hands-on experience evaluating LLM, agent or RAG systems.
• Comfortable with agreement metrics and spreadsheets.
• Fluent written English.

Contract

• Remote, hourly contract (independent contractor)
• Part-time, flexible: 10-40 hrs/week
• Remote

Pay: $60 - $90 / hour (USD). Paid in USD, weekly. Rate set after a short paid assessment.