AI Evaluation & Benchmarking Manager

PwC; Birmingham; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Empowered flexibility

Hybrid work model

Private medical cover

24/7 virtual GP

Volunteer days

Job summary

PwC is seeking a Data Scientist - AI Evaluation & Benchmarking Manager to join the AI Research team. You will shape model benchmarking and experimentation capabilities, driving evidence-based decisions for client engagements across PwC and clients.

Role involves building end-to-end benchmarking workflows, scalable evaluation frameworks, and production-grade experimentation infrastructure with a focus on robust, repeatable model assessment in fast-moving environments.

Qualifications

• Strong hands-on experience in Data Science concepts or LLM experimentation using structured evaluation frameworks.
• Proficient in Python, including asynchronous programming, multithreading and writing maintainable code.
• Experience deploying ML workloads to cloud platforms (Azure, AWS or GCP) with familiarity in CI/CD and containerisation.

Responsibilities

• Design and run end to end benchmarking workflows, from understanding client use cases, designing and running benchmarking strategies, and generating business ready insights
• Build scalable evaluation frameworks, metrics and pipelines, combining hands-on engineering with continuous review of academic literature to ensure our evaluation strategies reflect leading research and best practice.
• Develop, maintain and improve experimentation infrastructure, ensuring robustness and production grade engineering.
• Produce clear, client ready insights and support technical demos and deep dive sessions.

Skills

Data Science

Big Data

Python

LLM experimentation

Data Pipelines

Statistical Design

Leadership

Tools

Azure

AWS

GCP

Docker/ Podman

CI/CD

Job description

PwC is seeking a Data Scientist - AI Evaluation & Benchmarking Manager to join the AI Research team. You will shape model benchmarking and experimentation capabilities, driving evidence-based decisions for client engagements across PwC and clients.

Role involves building end-to-end benchmarking workflows, scalable evaluation frameworks, and production-grade experimentation infrastructure with a focus on robust, repeatable model assessment in fast-moving environments.