Director: AI Systems Reliability, Testing & Performance

Novartis; Greater London; Hybrid; GBP 100,000 - 186,000; Full time

Job summary

Novartis in London seeks a Director of AI Systems Reliability, Testing & Performance. You will define how AI systems across Development are evaluated, tested, validated, benchmarked and monitored throughout their lifecycle.

This senior leadership role owns the technical evidence for reliability, security and fitness-for-use, guiding evaluation, testing, observability and production-readiness across agentic AI, models, and digital twins.

Qualifications

• Senior AI leadership role within Data Science & AI.
• Responsible for defining evaluation, benchmarking, testing and validation methodologies for AI systems.

Responsibilities

• Define evaluation methodologies for AI systems, models, agents, digital twins, and simulation environments across Development.
• Establish benchmark suites, evaluation datasets, and testing harnesses used across AI initiatives.
• Define objective measures for quality, reliability, robustness, grounding, agent effectiveness, and task success.
• Ensure evaluation approaches remain scientifically rigorous, reproducible, and comparable across AI systems.
• Build a common evidence framework for assessing AI capabilities, limitations, and fitness-for-use.
• Establish approaches for hallucination testing, failure-mode analysis, robustness testing, and behavioral validation.
• Define validation methodologies for agentic systems, digital twins, simulation environments, and human-in-the-loop workflows.
• Develop production readiness criteria for AI systems operating in Development environments.
• Support technical validation activities required for GxP-relevant and regulated AI systems.
• Ensure AI systems are tested under realistic operating conditions and supported by objective validation evidence.
• Define how AI system reliability, degradation, drift, and operational performance are measured over time.
• Establish monitoring requirements and performance assessment frameworks across Development AI systems.
• Partner with engineering teams to ensure required evaluation, monitoring, tracing, and observability signals are available.
• Define methods for identifying, diagnosing, and assessing AI system failures.
• Promote evidence-based improvement of deployed AI capabilities.
• Establish portfolio-wide approaches for capturing technical evidence related to AI quality, reliability, robustness, and performance.
• Define standard reporting for benchmark results, validation findings, monitoring signals, and reliability assessments.
• Create transparency into AI system strengths, limitations, and failure modes.
• Generate technical evidence supporting validation, audit, inspection, and governance activities.
• Enable objective decisions on AI system readiness, reliability, and fitness-for-use.

Job description

Salary Range: £100,240.00 - £186,160.00

Job Description Summary

Director: AI Systems Reliability, Testing & Performance

#LI-Hybrid

Location: London

Novartis is unable to offer relocation support for this role: please only apply if this location is accessible for you.

The Director, AI Systems Reliability, Testing & Performance is a senior technical AI leadership role within Data Science & AI, responsible for defining how AI systems across Novartis Development are evaluated, tested, validated, benchmarked, monitored, and continuously improved throughout their lifecycle.

This role owns the technical evidence required to determine whether AI systems are reliable, robust, secure, and fit-for-use across Novartis Development. The role defines common approaches for AI evaluation, benchmarking, testing, validation, monitoring and production-readiness across agentic AI systems, predictive models, retrieval systems, digital twins, and other AI capabilities.

The Director provides technical leadership in AI evaluation science, reliability engineering, validation, adversarial testing, and performance assessment. Key areas of focus include model and agent evaluation, benchmarking, failure-mode analysis, drift detection, digital twin validation, AI red teaming, observability, traceability, and technical evidence generation supporting regulated and business-critical AI systems.

This is not a Governance, Product Management, PMO, or infrastructure operations role. Governance owns policies, risk frameworks, and approval processes. Product teams own roadmaps, adoption, and value realization. DDIT and engineering teams own platforms, infrastructure, and operational services.

Success means Development AI systems are supported by objective evidence demonstrating how they perform, where they fail, and whether they remain fit-for-use over time.

Job Description

Major Accountabilities

AI Evaluation & Benchmarking

Define evaluation methodologies for AI systems, models, agents, digital twins, and simulation environments across Development.

Establish benchmark suites, evaluation datasets, and testing harnesses used across AI initiatives.

Define objective measures for quality, reliability, robustness, grounding, agent effectiveness, and task success.

Ensure evaluation approaches remain scientifically rigorous, reproducible, and comparable across AI systems.

Build a common evidence framework for assessing AI capabilities, limitations, and fitness-for-use.

AI Testing & Validation

Establish approaches for hallucination testing, failure-mode analysis, robustness testing, and behavioral validation.

Define validation methodologies for agentic systems, digital twins, simulation environments, and human-in-the-loop workflows.

Develop production readiness criteria for AI systems operating in Development environments.

Support technical validation activities required for GxP-relevant and regulated AI systems.

Ensure AI systems are tested under realistic operating conditions and supported by objective validation evidence.

Reliability, Monitoring & Performance

Define how AI system reliability, degradation, drift, and operational performance are measured over time.

Establish monitoring requirements and performance assessment frameworks across Development AI systems.

Partner with engineering teams to ensure required evaluation, monitoring, tracing, and observability signals are available.

Define methods for identifying, diagnosing, and assessing AI system failures.

Promote evidence-based improvement of deployed AI capabilities.

AI Security & Adversarial Testing

Define approaches for AI red teaming, adversarial testing, and security evaluation.

Establish testing methodologies for prompt injection, jailbreaks, retrieval attacks, tool misuse, and agent manipulation scenarios.

Assess AI-specific vulnerabilities and resilience risks in collaboration with cybersecurity teams.

Ensure security testing forms part of AI system validation and production-readiness assessments.

Technical Evidence & AI Performance Intelligence

Establish portfolio-wide approaches for capturing technical evidence related to AI quality, reliability, robustness, and performance.

Define standard reporting for benchmark results, validation findings, monitoring signals, and reliability assessments.

Create transparency into AI system strengths, limitations, and failure modes.

Generate technical evidence supporting validation, audit, inspection, and governance activities.

Enable objective decisions on AI system readiness, reliability, and fitness-for-use.

Key Performance Indicators

• Percentage of AI systems meeting assurance standards.
• Coverage of evaluation and monitoring across AI portfolio.
• Reliability and availability of business-critical AI solutions.
• Audit and compliance readiness.
• Time to identify and resolve AI performance issues.
• Percentage of AI initiatives with measurable business outcomes.
• Leadership confidence in AI performance reporting.