Evaluation Engineer

Freshfields; Greater London; On-site; GBP 60,000 - 80,000; Full time

Job summary

Freshfields is hiring for an AI evaluation role focused on testing AI applications within legal and business contexts.

You will implement automated evaluation cases, prepare test datasets with legal specialists, and run comprehensive evaluations of answer accuracy, retrieval relevance and instruction following. The role emphasizes reproducibility, evidence-based defect analysis and collaboration with senior engineers.

Qualifications

• Software development experience using Python, Java or C#.
• Foundational understanding of LLMs, capabilities, limitations and failure modes.
• Ability to use coding agents for development, testing and analysis with best practices.
• Working knowledge of APIs, Git, automated testing and CI/CD tools (Jira/Jenkins).
• Curiosity and ability to learn in a rapidly evolving AI landscape with guidance.

Responsibilities

• Build automated evaluation cases and supporting scripts.
• Prepare and maintain test datasets with legal specialists, ensuring confidentiality.
• Run evaluations on answer accuracy, citations, retrieval relevance and escalation criteria.
• Execute functional and integration tests for AI workflows and error handling.
• Review outputs, reproduce issues and document supporting evidence for defects.
• Compare results across model, prompt and application versions and report changes.

Skills

Python

Java

C#

APIs

CI/CD

Education

Bachelor's degree in Computer Science

Tools

Git

Jira

Jenkins

Job description

Build and run tests that assess the quality and reliability of the firm’s AI applications. Help maintain evaluation datasets, investigate failures and provide clear evidence of how AI systems perform on legal and business tasks.

Key responsibilities

• Implement automated evaluation cases and supporting scripts using established frameworks and guidance from senior engineers.
• Prepare and maintain test datasets, source documents and reference examples with legal specialists. Follow requirements for confidentiality, permitted data use and version control.
• Run evaluations of answer accuracy, completeness, citations, retrieval relevance, instruction following and appropriate escalation. Apply agreed scoring criteria and flag cases requiring expert judgement.
• Execute functional and integration tests for AI workflows, including document ingestion, tool calls, structured outputs and error handling, in coordination with application QA.
• Review outputs and execution traces to identify failures. Reproduce issues, document supporting evidence and help distinguish data, retrieval, model and application defects.
• Compare results across model, prompt and application versions. Report changes accurately, including uncertainty and limitations, rather than relying solely on headline scores.
• Support the calibration of automated graders by comparing their scores with human reviews and investigating inconsistencies.
• Maintain regression suites, contribute to CI/CD integration and turn approved production failure examples into new test cases.
• undefined

Required skills and experience

• Experience developing software demonstrated through employment, placements or substantial projects using programming languages such as Python, Java or C#.
• Foundational conceptual understanding of large language models, including their capabilities, limitations and common failure modes, and an ability to apply that understanding when developing evaluation criteria, test cases and acceptance thresholds with guidance from senior engineers.
• Ability to use coding agents effectively to support development, testing and analysis, with awareness of best practices for prompting, context management, validation, security, responsible use of generated code and when to seek human review.
• Working knowledge of APIs, Git, automated testing and software development lifecycle practices and CI/CD tools such as Jira and Jenkins.
• Demonstrated curiosity and ability to learn in a rapidly evolving AI landscape, assess new models, tools and evaluation techniques, and apply relevant learning pragmatically with appropriate guidance.
• Foundational understanding of statistics and measurement, including sampling, precision and recall, together with strong analytical skills, attention to detail and the ability to document issues clearly and seek technical or legal guidance when needed.
• Practical experience evaluating AI applications is a plus but is not required, given the newness of the field; relevant academic, internship or project experience will also be considered.

Inclusion

Freshfields is an equal opportunities employer and all applications received by the firm will be considered by the firm on the basis of their merit alone and we welcome applications from all suitably qualified individuals regardless of background. All offers of employment will be conditional on the candidate having/securing the right to work in the UK and providing the firm with evidence of that right (as required by the Immigration, Asylum and Nationality Act 2006) prior to employment commencing.

Freshfields is a Ban the Box employer. We ask applicants to disclose criminal convictions only when a conditional job offer is made. A conviction does not automatically lead to withdrawal of the offer: we make decisions on a case by case basis and take a number of factors into account (eg the role you are applying for and the circumstances of the offence). You would have the opportunity to discuss the matter with us before we make a decision.