Senior AI Evaluation Engineer

Incubator for Artificial Intelligence; West of England; Hybrid; GBP 63,000 - 87,000; Full time

Benefits offered by this job

Hybrid working locations (London/Manch

Dental insurance

Cycle to work

25+ days leave

Parental leave up to 39 weeks

Technical allowance up to £15,600

Employer pension 28.97%

Job summary

Incubator for AI (i.AI) seeks two Senior AI Evaluation Engineers to lead model-level evaluation of accuracy, bias and safety. You will design evaluation pipelines, develop advanced methods, and work with product and domain experts across London, Manchester or Bristol. Strong Python skills and mentoring experience preferred.

The role offers hybrid working, access to frontier AI, and impact on public services with outputs published through reports and external publications.

Qualifications

• Broad experience evaluating generative and predictive AI systems, including LLM-as-a-judge and tools such as pydantic evals, deepeval or inspect.
• Extensive knowledge of experimental design, statistical hypothesis testing and quantitative research methods (particularly those in the Magenta Book).
• High proficiency in Python and strong software practices, including git, CI/CD, unit testing and pipeline orchestration tools like DVC or snakemake.
• The ability to communicate technical work to broad audiences through reports, presentations and web-based tools.
• Experience mentoring junior colleagues.

Responsibilities

• Leading the design of evaluation pipelines and architecting product-wide evaluation suites that integrate with engineering and CI/CD workflows
• Developing complex and exploratory methods to assess accuracy, bias, safety and model performance
• Working with product, engineering and domain experts to shape evaluation across the product lifecycle and inform technical decisions
• Turning complex results into clear recommendations and leading communication of findings through reports, presentations and external publications
• Reviewing and supporting other engineers’ work to raise standards and build evaluation capability across the team

Skills

Generative AI evaluation

LLM evaluation

Python

Experimental design

Communication

Mentoring

Tools

pydantic evals

deepeval

inspect

Git

CI/CD

DVC

Snakemake

Job description

Senior AI Evaluation Engineer (Data Scientist) - London/Manchester/Bristol

The Incubator for AI (i.AI) is a team of around 80 builders and operators using frontier AI to improve public services across healthcare, education, planning and parliament. We work at startup pace, with the backing of the Prime Minister and No10, on problems that affect millions of people.

Our Impact and Evaluation team embeds within product teams to build rigorous evaluation into every stage of development, shaping product decisions and showing whether our products improve public services. We’re now hiring two Senior AI Evaluation Engineers to focus on model-level evaluation of accuracy, bias and safety.

What will I be doing?

• Leading the design of evaluation pipelines and architecting product-wide evaluation suites that integrate with engineering and CI/CD workflows
• Developing complex and exploratory methods to assess accuracy, bias, safety and model performance
• Working with product, engineering and domain experts to shape evaluation across the product lifecycle and inform technical decisions
• Turning complex results into clear recommendations and leading communication of findings through reports, presentations and external publications
• Reviewing and supporting other engineers’ work to raise standards and build evaluation capability across the team

What will I need?

You don’t need to tick every box. You’ll likely bring some of the following:

• Broad experience evaluating generative and predictive AI systems, including LLM-as-a-judge and tools such as pydantic evals, deepeval or inspect
• Extensive knowledge of experimental design, statistical hypothesis testing and quantitative research methods (particularly those in the Magenta Book)
• High proficiency in Python and strong software practices, including git, CI/CD, unit testing and pipeline orchestration tools like DVC or snakemake
• The ability to communicate technical work to broad audiences through reports, presentations and web-based tools
• Experience mentoring junior colleagues

What’s in it for me?

• Career-defining work. Projects backed by No10 that apply AI to public services affecting millions, alongside talented, mission-driven colleagues
• The tools to ship. Frontier models, ample compute, access to experts in national security, policy and AI research, and full operational and delivery support
• Early ownership of important products in small teams, 5 L&D days a year, and funding for learning, conferences and collaborations
• £63,165–£86,640 in London (£58,275–£81,885 nationally), including a technical allowance of up to £15,600, plus a 28.97% employer pension contribution
• Flexibility and benefits. Hybrid working from London, Manchester or Bristol, 25+ days’ leave, up to 39 weeks’ full-pay parental leave, and perks including cycle to work and dental insurance