Senior AI Evaluation Engineer

Incubator for Artificial Intelligence; Greater London; Hybrid; GBP 63,000 - 87,000; Full time

Benefits offered by this job

Hybrid work from multiple cities

Generous leave

Pension contribution

Cycle to work

Dental insurance

Job summary

The Incubator for AI (i.AI) seeks two Senior AI Evaluation Engineers to focus on model-level evaluation of accuracy, bias and safety for frontier AI applications in public services. You will design evaluation pipelines, work with cross-functional teams, and communicate findings through reports and external publications.

Ideal candidates have strong Python skills, experience with experimental design, and a track record mentoring colleagues.

Qualifications

• Experience evaluating generative and predictive AI systems (LLM-as-a-judge; pydantic evals, deepeval, inspect).
• Strong knowledge of experimental design and quantitative research methods (Magenta Book).
• High proficiency in Python and software practices (git, CI/CD, unit testing, DVC or snakemake).
• Ability to communicate technical work to broad audiences via reports and presentations.
• Experience mentoring junior colleagues.

Responsibilities

• Leading the design of evaluation pipelines and product-wide evaluation suites.
• Developing methods to assess accuracy, bias, safety and model performance.
• Collaborating with product, engineering and domain experts across the lifecycle.
• Translating results into clear recommendations and communicating findings.
• Reviewing work to raise standards and build evaluation capability.

Skills

Generative AI evaluation

Python

CI/CD

Experiment design

Communication

Mentoring

Tools

pydantic evals

deepeval

inspect

DVC

Snakemake

Job description

Senior AI Evaluation Engineer (Data Scientist) - London/Manchester/Bristol

The Incubator for AI (i.AI) is a team of around 80 builders and operators using frontier AI to improve public services across healthcare, education, planning and parliament. We work at startup pace, with the backing of the Prime Minister and No10, on problems that affect millions of people.

Our Impact and Evaluation team embeds within product teams to build rigorous evaluation into every stage of development, shaping product decisions and showing whether our products improve public services. We’re now hiring two Senior AI Evaluation Engineers to focus on model-level evaluation of accuracy, bias and safety.

What will I be doing?

• Leading the design of evaluation pipelines and architecting product-wide evaluation suites that integrate with engineering and CI/CD workflows
• Developing complex and exploratory methods to assess accuracy, bias, safety and model performance
• Working with product, engineering and domain experts to shape evaluation across the product lifecycle and inform technical decisions
• Turning complex results into clear recommendations and leading communication of findings through reports, presentations and external publications
• Reviewing and supporting other engineers’ work to raise standards and build evaluation capability across the team

What will I need?

You don’t need to tick every box. You’ll likely bring some of the following:

• Broad experience evaluating generative and predictive AI systems, including LLM-as-a-judge and tools such as pydantic evals, deepeval or inspect
• Extensive knowledge of experimental design, statistical hypothesis testing and quantitative research methods (particularly those in the Magenta Book)
• High proficiency in Python and strong software practices, including git, CI/CD, unit testing and pipeline orchestration tools like DVC or snakemake
• The ability to communicate technical work to broad audiences through reports, presentations and web-based tools
• Experience mentoring junior colleagues

What’s in it for me?

• Career-defining work. Projects backed by No10 that apply AI to public services affecting millions, alongside talented, mission-driven colleagues
• The tools to ship. Frontier models, ample compute, access to experts in national security, policy and AI research, and full operational and delivery support
• Early ownership of important products in small teams, 5 L&D days a year, and funding for learning, conferences and collaborations
• £63,165–£86,640 in London (£58,275–£81,885 nationally), including a technical allowance of up to £15,600, plus a 28.97% employer pension contribution
• Flexibility and benefits. Hybrid working from London, Manchester or Bristol, 25+ days’ leave, up to 39 weeks’ full-pay parental leave, and perks including cycle to work and dental insurance