Senior Clinical AI Evaluation & Responsible Deployment Lead

AstraZeneca; Barcelona; Hybrid; EUR 110,000 - 140,000; Full time

Benefits offered by this job

In-office collaboration in Barcelona

Flexible work arrangement

Job summary

AstraZeneca is hiring an Associate Director, Clinical AI Evaluation and Responsible Deployment to establish evidence systems that determine usefulness, reliability, safety, and scale of clinical AI. You will collaborate with clinical, engineering, IT, and data teams to shape validation strategies and governance across the product lifecycle.

You will build evaluation harnesses, define acceptance thresholds, and ensure audit-ready evidence aligned with GCP/GxP.

Qualifications

• PhD in Machine Learning, Computer Science, Statistics, Biostatistics, Biomedical Informatics, Computational Biology, or a related quantitative discipline; or an MD, master’s degree, or equivalent experience with a strong computational record.
• 4 to 7 years of post-PhD (or equivalent) experience evaluating, validating, or deploying AI/ML systems in healthcare, life sciences, clinical research, or another safety-or-evidence-critical environment.
• Hands-on coding ability in Python and SQL, including experience building automated evaluation pipelines, analysing large datasets, writing tests, and working in version-controlled environments.
• Strong grounding in experimental design, statistical inference, uncertainty, error analysis, and measurement reliability.
• Experience evaluating LLMs or agentic systems, including retrieval, tool use, structured outputs, multi-step workflows, robustness, and human-in-the-loop performance.
• Ability to construct useful evaluation approaches when labels are noisy, expert opinions differ, or outcomes are delayed.

Responsibilities

• Define and implement the evaluation strategy for clinical AI models and agents across development, release, monitoring, and change control.
• Build evaluation harnesses, curated test sets, simulation environments, automated regression suites, and analysis pipelines in Python and related technologies.
• Translate expert judgment from CRAs, medical monitors, clinical scientists, statisticians, and operations leaders into task definitions, scoring rubrics, error taxonomies, and clinically meaningful acceptance thresholds.
• Evaluate complete workflows—not only model outputs—including retrieval quality, tool selection, orchestration, source fidelity, abstention, human hand-offs, latency, and downstream operational impact.
• Design approaches for noisy, sparse, delayed, or expert-dependent ground truth, including adjudication, inter-rater agreement, challenge sets, prospective studies, and post-deployment surveillance.
• Lead failure analysis and red-teaming for hallucination, unsupported claims, automation bias, data leakage, prompt injection, subgroup performance, and unsafe workflow behaviour.
• Establish traceability from intended use and user need through requirements, test evidence, and release decisions, including how model, prompt, tool, data, and workflow changes are assessed, monitored, and revalidated throughout the product lifecycle.

Skills

Python

SQL

Experimental design

Statistical inference

LLM evaluation

Git

Education

PhD in Machine Learning or related field

Tools

Jupyter

Job description

AstraZeneca is hiring an Associate Director, Clinical AI Evaluation and Responsible Deployment to establish evidence systems that determine usefulness, reliability, safety, and scale of clinical AI. You will collaborate with clinical, engineering, IT, and data teams to shape validation strategies and governance across the product lifecycle.

You will build evaluation harnesses, define acceptance thresholds, and ensure audit-ready evidence aligned with GCP/GxP.