AI Evaluation Engineer — QA for AI Workflows

Freshfields; Greater London; On-site; GBP 60,000 - 80,000; Full time

Job summary

Freshfields is hiring for an AI evaluation role focused on testing AI applications within legal and business contexts.

You will implement automated evaluation cases, prepare test datasets with legal specialists, and run comprehensive evaluations of answer accuracy, retrieval relevance and instruction following. The role emphasizes reproducibility, evidence-based defect analysis and collaboration with senior engineers.

Qualifications

• Software development experience using Python, Java or C#.
• Foundational understanding of LLMs, capabilities, limitations and failure modes.
• Ability to use coding agents for development, testing and analysis with best practices.
• Working knowledge of APIs, Git, automated testing and CI/CD tools (Jira/Jenkins).
• Curiosity and ability to learn in a rapidly evolving AI landscape with guidance.

Responsibilities

• Build automated evaluation cases and supporting scripts.
• Prepare and maintain test datasets with legal specialists, ensuring confidentiality.
• Run evaluations on answer accuracy, citations, retrieval relevance and escalation criteria.
• Execute functional and integration tests for AI workflows and error handling.
• Review outputs, reproduce issues and document supporting evidence for defects.
• Compare results across model, prompt and application versions and report changes.

Skills

Python

Java

C#

APIs

CI/CD

Education

Bachelor's degree in Computer Science

Tools

Git

Jira

Jenkins

Job description

Freshfields is hiring for an AI evaluation role focused on testing AI applications within legal and business contexts.

You will implement automated evaluation cases, prepare test datasets with legal specialists, and run comprehensive evaluations of answer accuracy, retrieval relevance and instruction following. The role emphasizes reproducibility, evidence-based defect analysis and collaboration with senior engineers.