Remote AI Evaluation QA Lead

Vettedlance; United States; Remote; USD 83.000 - 124.000; Teilzeit

Zusammenfassung

Vettedlance is seeking an experienced QA lead to run quality for AI evaluation projects, including rubric design, reviewer calibration and tracking agreement for agentic and RAG systems.

You will own QA for agentic-workflow, write and refine rubrics and gold sets, calibrate reviewers and monitor inter-annotator agreement, audit samples, and report quality trends. Remote, hourly contract, part-time, 10-40 hrs/week.

Qualifikationen

• 4+ years owning QA on annotation, evaluation or software testing.
• Hands-on experience evaluating LLM, agent or RAG systems.
• Comfortable with agreement metrics and spreadsheets.
• Fluent written English.

Aufgaben

• Own QA for agentic-workflow and RAG/conversational evaluation projects.
• Write and refine rubrics and gold sets.
• Calibrate reviewers and track inter-annotator agreement (kappa, alpha).
• Audit samples and report quality trends.

Kenntnisse

QA leadership

Rubric design

Reviewer calibration

Inter-annotator agreement

Agreement tracking

LLM evaluation

RAG systems

Jobbeschreibung

Vettedlance is seeking an experienced QA lead to run quality for AI evaluation projects, including rubric design, reviewer calibration and tracking agreement for agentic and RAG systems.

You will own QA for agentic-workflow, write and refine rubrics and gold sets, calibrate reviewers and monitor inter-annotator agreement, audit samples, and report quality trends. Remote, hourly contract, part-time, 10-40 hrs/week.