Head of AI Reliability, Testing & Performance

Novartis UK; Greater London; On-site; GBP 100,000 - 186,000; Full time

Benefits offered by this job

Flexible and hybrid working options

Parental leave (14 weeks minimum)

Job summary

Novartis UK is seeking a Director of AI Systems Reliability, Testing & Performance in London to lead cross‑functional AI testing, evaluation and validation across Development. You will define evidence frameworks, benchmark datasets and production readiness criteria for regulated AI systems.

In this senior role, you will drive observability, drift detection, and performance intelligence, collaborating with engineering to ensure robust, secure, and fit‑for‑use AI capabilities across the portfolio.

Qualifications

• 10+ years in AI, machine learning, software engineering, quality, risk, validation, or technology governance.
• Experience deploying or overseeing business‑critical AI systems.
• Experience establishing governance, controls, quality standards, or operational frameworks.

Responsibilities

• Define evaluation methodologies for AI systems, models, agents, digital twins, and simulation environments.
• Establish benchmark suites, evaluation datasets, and testing harnesses used across AI initiatives.
• Define objective measures for quality, reliability, robustness, grounding, agent effectiveness, and task success.
• Ensure evaluation approaches remain scientifically rigorous, reproducible, and comparable across AI systems.
• Build a common evidence framework for assessing AI capabilities, limitations, and fitness-for-use.
• Establish approaches for hallucination testing, failure‑mode analysis, robustness testing, and behavioral validation.
• Define validation methodologies for agentic systems, digital twins, simulation environments, and human‑in‑the‑loop workflows.
• Develop production readiness criteria for AI systems operating in Development environments.
• Support technical validation activities required for GxP-relevant and regulated AI systems.
• Ensure AI systems are tested under realistic operating conditions and supported by objective validation evidence.
• Define how AI system reliability, degradation, drift, and operational performance are measured over time.
• Establish monitoring requirements and performance assessment frameworks across Development AI systems.
• Partner with engineering teams to ensure required evaluation, monitoring, tracing, and observability signals are available.
• Define methods for identifying, diagnosing, and assessing AI system failures.
• Promote evidence‑based improvement of deployed AI capabilities.
• Define approaches for AI red teaming, adversarial testing, and security evaluation.
• Establish testing methodologies for prompt injection, jailbreaks, retrieval attacks, tool misuse, and agent manipulation scenarios.
• Assess AI‑specific vulnerabilities and resilience risks in collaboration with cybersecurity teams.
• Ensure security testing forms part of AI system validation and production‑readiness assessments.
• Establish portfolio‑wide approaches for capturing technical evidence related to AI quality, reliability, robustness, and performance.
• Define standard reporting for benchmark results, validation findings, monitoring signals, and reliability assessments.
• Create transparency into AI system strengths, limitations, and failure modes.
• Generate technical evidence supporting validation, audit, inspection, and governance activities.
• Enable objective decisions on AI system readiness, reliability, and fitness‑for‑use.

Skills

AI leadership

Reliability engineering

Validation

Governance & risk management

Performance measurement

Job description

Novartis UK is seeking a Director of AI Systems Reliability, Testing & Performance in London to lead cross‑functional AI testing, evaluation and validation across Development. You will define evidence frameworks, benchmark datasets and production readiness criteria for regulated AI systems.

In this senior role, you will drive observability, drift detection, and performance intelligence, collaborating with engineering to ensure robust, secure, and fit‑for‑use AI capabilities across the portfolio.