Quality Systems Lead: AI Data Quality & Audit Automation

Encord; Greater London; On-site; GBP 110,000 - 140,000; Full time

Benefits offered by this job

Competitive salary

Equity

London office

25 days leave

Learning budget

Travel for UK/Europe

Team lunches

Socials & offsites

Job summary

Encord is hiring a Quality Systems Lead to own how we measure the quality of the human data we deliver to frontier AI labs, physical AI companies and enterprise AI teams. You will build automated evaluation pipelines, manage an audit team of around ten specialists based in India, and turn audit output into a scalable ground truth that trains the automated layer.

You will define the quality standard for every data type, write rubrics with edge cases, and set acceptance criteria in collaboration

Qualifications

• 4+ years owning both technical and operational outcomes in a data, AI or service delivery environment where quality was measured rather than asserted.
• Hands-on Python and SQL. You build the analysis and the tooling rather than specify it for someone else.
• Practical experience applying models to a quality or evaluation problem — LLM-as-judge, model-assisted QA, automated evaluation, anomaly detection or classifier-based screening.
• Sampling methodology and agreement statistics (Cohen's and Fleiss' kappa, F1 against ground truth) applied to real production data.
• Experience hiring, training and managing a team, ideally an audit, review or QA team, and ideally distributed.
• Track record of building a quality framework or function, including the reporting leadership and customers run on.
• Bonus: direct experience of annotation, evaluation or model-training workflows, and of what frontier AI labs accept as evidence of quality.
• Bonus: a STEM degree, or a background in data science or research engineering
• Bonus: multilingual delivery and linguistic quality assessment

Responsibilities

• Build automated dataset quality evaluation and root-cause detection — model-assisted and LLM-as-judge screening, agreement analysis at scale, anomaly and drift detection across annotation output
• Hire, train, calibrate and manage a dedicated audit team of around ten specialists based in our India operation, held to inter-rater agreement and catch rate rather than volume audited
• Turn audit output into labelled ground truth that trains and validates the automated layer, and manage the ratio of automated to manual coverage deliberately over time
• Own the quality standard for every data type we deliver — written rubrics with worked edge cases, golden sets, and acceptance criteria agreed with the customer, alongside the Special Projects lead, before the first batch ships
• Build scoring systems that rank annotator and reviewer performance and feed routing, staffing and offboarding decisions
• Set the pass thresholds that certification gates on, so nobody works a project queue without having demonstrated they meet the standard
• Report quality KPIs to leadership, and into the reporting our Special Projects leads take to customers: accuracy against client spec, inter-annotator agreement, rework rate, cost of rework, and coverage
• Work with Project Management on remediation — you produce the measurement and the diagnosis, production owns fixing the project, and the standard stays independent of the people being measured
• Own the unit economics of quality: cost per audited unit, and the coverage you buy per pound spent
• Partner with Product and Engineering to bring quality measurement into Encord platform as native capability

Skills

Python

SQL

Data quality

Quality assurance

Auditing

Statistical methods

Education

STEM degree

Tools

LLM-based evaluation

Data automation pipelines

Job description

Encord is hiring a Quality Systems Lead to own how we measure the quality of the human data we deliver to frontier AI labs, physical AI companies and enterprise AI teams. You will build automated evaluation pipelines, manage an audit team of around ten specialists based in India, and turn audit output into a scalable ground truth that trains the automated layer.

You will define the quality standard for every data type, write rubrics with edge cases, and set acceptance criteria in collaboration