AI Evaluation Engineer - Quality & Testing

Freshfields Bruckhaus Deringer; England; On-site; GBP 65,000 - 95,000; Full time

Job summary

Freshfields Bruckhaus Deringer seeks a QA Engineer to build and run tests for AI applications, evaluating accuracy, citations, and retrieval relevance. You will prepare evaluation datasets with legal specialists, ensure confidentiality, and follow stringent data-use rules.

You will work with senior engineers to implement automated test cases, maintain regression suites, and analyse failures across model and application versions, contributing to robust AI systems for legal tasks.

Qualifications

• Five years’ software development experience with Python, Java or C#.
• Foundational understanding of large language models and their failure modes.
• Ability to use coding agents and best practices for prompting and security.
• Experience with APIs, Git, automated testing and CI/CD tools.
• Curiosity to learn in a rapidly evolving AI landscape.

Responsibilities

• Develop automated evaluation cases and supporting scripts.
• Prepare test datasets with legal specialists and ensure data confidentiality.
• Run evaluations of answer accuracy, citations, and retrieval relevance.
• Execute functional tests for AI workflows with QA collaboration.
• Review outputs to identify failures and reproduce issues.
• Compare results across model, prompt and application versions.
• Assist calibration of automated graders with human reviews.
• Maintain regression suites and CI/CD integrations.

Skills

Python

Java

C#

LLMs understanding

Testing frameworks

CI/CD concepts

Prompting best practices

Documentation & debugging

Tools

Git

Jira

Jenkins

Job description

Freshfields Bruckhaus Deringer seeks a QA Engineer to build and run tests for AI applications, evaluating accuracy, citations, and retrieval relevance. You will prepare evaluation datasets with legal specialists, ensure confidentiality, and follow stringent data-use rules.

You will work with senior engineers to implement automated test cases, maintain regression suites, and analyse failures across model and application versions, contributing to robust AI systems for legal tasks.