AI Evaluation Quality Architect

Inquigent Ltd.; United Kingdom; Remote; GBP 65,000 - 95,000; Full time

Benefits offered by this job

Remote-first

Learning budget

Equipment provided

Job summary

Inquigent Ltd. is seeking an AI Evaluation Quality Lead to own and uphold the quality standards behind assessment tasks, answer keys, reviewer guidance, and feedback for managed evaluation projects.

You will work hands-on in LLM evaluation, data annotation quality, and related areas to ensure consistent, defensible decisions. You will design rubrics, calibrate reviewers, review advisory outputs, and develop actionable guidance.

Qualifications

• Hands-on experience in LLM evaluation, data annotation quality, search or ads quality rating, or similar.
• Ability to write a rubric that two reviewers apply the same way.
• Analytical and precise; comfortable with sampling and simple metrics.

Responsibilities

• Design and maintain assessment content, rubrics and reviewer guidance so decisions are consistent and defensible
• Set and check quality standards on managed projects: sampling, calibration and feedback loops
• Review AI advisory grading output and decide where it helps and where it misleads
• Write development guidance members can act on after a retry decision
• Work with the Talent Operations Lead on reviewer training and calibration sessions

Skills

LLM evaluation experience

Data annotation quality

Analytical and precise

Sampling and metrics

Rubric writing

Job description

Inquigent Ltd. is seeking an AI Evaluation Quality Lead to own and uphold the quality standards behind assessment tasks, answer keys, reviewer guidance, and feedback for managed evaluation projects.

You will work hands-on in LLM evaluation, data annotation quality, and related areas to ensure consistent, defensible decisions. You will design rubrics, calibrate reviewers, review advisory outputs, and develop actionable guidance.