Senior Scientist, Generative AI Evaluation & Standards

Johnson Johnson; Madrid; Híbrido; EUR 85.000 - 110.000; Jornada completa

Descripción de la vacante

Johnson & Johnson is seeking a Senior Scientist to lead the evaluation of Generative AI solutions for pharma R&D within the EQS function in Data Science and Digital Health. You will design, build, and run evaluation methods to ensure AI systems deliver quality, defensible results across literature review, evidence synthesis, and decision support.

You will author rubrics, curate benchmark datasets, validate AI judges, and produce comparable readouts to inform release decisions.

Formación

• Master's degree in a related field required; PhD preferred.
• 6+ years in AI/ML evaluation or data science (Master's) or 3+ years with a PhD.
• Experience designing and running evaluation frameworks and benchmarks for AI/ML systems.
• Hands-on work with generative AI, LLMs, retrieval-augmented generation, and prompt engineering.

Responsabilidades

• Design, build, and maintain automated evaluation pipelines for LLM quality and RAG performance.
• Author rubrics and curate datasets; maintain reusable evaluation assets.
• Validate AI judges and run model, prompt, retriever, and agent benchmarks.
• Analyze failure patterns and provide actionable recommendations.
• Develop evaluation criteria with scientific and regulatory partners.
• Design evaluation methods for scientific reasoning and evidence synthesis.
• Build tooling to enable self-serve evaluation by other teams.

Conocimientos

AI/ML evaluation

Python

LLM APIs

Evaluation frameworks

Data science

Prompt engineering

Vector databases

Educación

Master's degree

PhD preferred

Herramientas

Evaluation harnesses

Embedding models

Vector databases

LLM APIs

Descripción del empleo

Johnson & Johnson is seeking a Senior Scientist to lead the evaluation of Generative AI solutions for pharma R&D within the EQS function in Data Science and Digital Health. You will design, build, and run evaluation methods to ensure AI systems deliver quality, defensible results across literature review, evidence synthesis, and decision support.

You will author rubrics, curate benchmark datasets, validate AI judges, and produce comparable readouts to inform release decisions.