AI Data Quality & Audit Lead

Encord; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Competitive compensation and equity

London office culture

25 days holiday + UK holidays

Learning & development budget

UK/Europe travel

Company lunches

Team socials & offsites

Job summary

Encord is hiring a Quality Systems Lead in London to own and scale how we measure data quality across AI data types. You will build automated evaluation, manage a ~10-person audit team in India, and turn audit outputs into gold standards for training the automated layer.

You’ll define quality rubrics, thresholds, and KPI reporting for customers while shaping product-facing quality capabilities. You will drive automation, establish standardized practices, and partner with Product and Engineering

Qualifications

• 4+ years owning both technical and operational outcomes in a data/AI or service delivery environment where quality was measured.
• Hands-on Python and SQL. You build the analysis and the tooling rather than specify it for someone else.
• Practical experience applying models to a quality or evaluation problem — LLM-as-judge, model-assisted QA, automated evaluation, anomaly detection or classifier-based screening.
• Sampling methodology and agreement statistics (e.g. Cohen's and Fleiss' kappa, F1 against ground truth) applied to real production data.
• Experience hiring, training and managing a team, ideally an audit, review or QA team, and ideally distributed.
• Track record of building a quality framework or function, including the reporting to leadership and customers.
• Bonus: direct experience of annotation, evaluation or model-training workflows, and of what frontier AI labs accept as evidence of quality.
• Bonus: a STEM degree, or a background in data science or research engineering.
• Bonus: multilingual delivery and linguistic quality assessment

Responsibilities

• Build automated dataset quality evaluation and root-cause detection — model-assisted and LLM-as-judge screening, agreement analysis at scale, anomaly and drift detection across annotation output
• Hire, train, calibrate and manage a dedicated audit team of around ten specialists based in our India operation, held to inter-rater agreement and catch rate rather than volume audited
• Turn audit output into labelled ground truth that trains and validates the automated layer, and manage the ratio of automated to manual coverage deliberately over time
• Own the quality standard for every data type we deliver — written rubrics with worked edge cases, golden sets, and acceptance criteria agreed with the customer, alongside the Special Projects lead, before the first batch ships
• Build scoring systems that rank annotator and reviewer performance and feed routing, staffing and offboarding decisions
• Set the pass thresholds that certification gates on, so nobody works a project queue without having demonstrated they meet the standard
• Report quality KPIs to leadership, and into the reporting our Special Projects leads take to customers: accuracy against client spec, inter-annotator agreement, rework rate, cost of rework, and coverage
• Work with Project Management on remediation — you produce the measurement and the diagnosis, production owns fixing the project, and the standard stays independent of the people being measured
• Own the unit economics of quality: cost per audited unit, and the coverage you buy per pound spent
• Partner with Product and Engineering to bring quality measurement into Encord platform as native capability

Skills

Statistical literacy

Leadership

Systems thinking

Technical writing

Education

STEM degree

Tools

Python

SQL

LLM-based evaluation

Model-assisted QA

Job description

Encord is hiring a Quality Systems Lead in London to own and scale how we measure data quality across AI data types. You will build automated evaluation, manage a ~10-person audit team in India, and turn audit outputs into gold standards for training the automated layer.

You’ll define quality rubrics, thresholds, and KPI reporting for customers while shaping product-facing quality capabilities. You will drive automation, establish standardized practices, and partner with Product and Engineering