Senior Scientist - GenAI Evaluation

Johnson & Johnson Innovative Medicine; Cornellà de Llobregat; Híbrido; EUR 55.000 - 88.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Annual bonus

Vacation days

Parental leave

Bereavement leave

Well-being reimbursement

Insurance plans

Descripción de la vacante

Johnson & Johnson is building Generative AI solutions to support pharmaceutical R&D and decision workflows. This hybrid role in Madrid or Barcelona involves designing evaluation frameworks for AI quality, safety, and scientific accuracy, collaborating with clinical and regulatory partners across Europe.

You will lead automated pipelines, curate datasets with domain experts, and benchmark models, prompts, and retrievers to ensure measurable, traceable, and defensible AI outputs in a regulated

Formación

• Master's degree in AI/ML or related field; PhD preferred.
• 6+ years in AI/ML evaluation or data science (Master's) or 3+ years with a PhD.
• Experience designing evaluation frameworks and scientific benchmarks.
• Hands-on with generative AI, LLMs, RAG, and prompt engineering.
• Strong Python skills and familiarity with modern AI/ML tooling.

Responsabilidades

• Design, build, and maintain automated evaluation pipelines for LLM quality, RAG performance, agent reliability, safety, and scientific accuracy.
• Author evaluation rubrics and scoring criteria, curate datasets with domain experts, and maintain reusable evaluation assets.
• Validate AI judges against human expert agreement and run benchmarks for models, prompts, retrievers, and agents.
• Analyze failure patterns and turn findings into actionable recommendations.
• Develop therapeu­tic-area specific evaluation criteria with scientific, clinical, and regulatory partners.
• Design evaluation methods for scientific reasoning and evidence synthesis.
• Build evaluation tooling and reusable patterns for cross-team use.

Conocimientos

Evaluation frameworks

Prompt engineering

Generative AI

Python

AI/ML tooling

Educación

Master's degree in AI/ML, CS, Data Science, Computational Biology, Bioinformatics, Biomedical Eng.

PhD preferred

Herramientas

Python

LLM APIs

Vector databases

Evaluation harnesses

Descripción del empleo

At Johnson & Johnson,we believe health is everything. Our strength in healthcare innovation empowers us to build aworld where complex diseases are prevented, treated, and cured,where treatments are smarter and less invasive, andsolutions are personal.Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity.Learn more at jnj.com.

As guided by Our Credo, Johnson & Johnson is responsible to our employees who work with us throughout the world. We provide an inclusive work environment where each person is considered as an individual. At Johnson & Johnson, we respect the diversity and dignity of our employees and recognize their merit.

Job Function

Job Sub Function

Job Category

All Job Posting Locations:

Cornellà de Llobregat, Barcelona, Spain, Madrid, Spain

Job Description

At J&J we are building Generative AI solutions to support pharmaceutical R&D — literature review, evidence synthesis, document Q&A, therapeutic area knowledge search, translational science workflows, and R&D decision support. These systems need to be evaluated before teams rely on them in scientific workflows. In pharma, a useful AI response depends on the question, user, source material, therapeutic area, and risk of error — so quality must be measurable, repeatable, traceable, and scientifically defensible.

Key Responsibilities

• Design, build, and maintain automated evaluation pipelines for LLM quality, RAG performance, agent reliability, safety, and scientific accuracy.
• Author evaluation rubrics and scoring criteria, curate golden and synthetic datasets with domain experts, and maintain our registry of reusable evaluation assets.
• Validate AI judges against human expert agreement and run model, prompt, retriever, and agent benchmarks that produce standardized quality readouts.
• Analyze failure patterns — hallucination, unsupported claims, weak traceability — and turn findings into actionable recommendations.
• Develop therapeutic‑area‑specific evaluation criteria with scientific, clinical, and regulatory partners, refining them based on real‑world feedback.
• Design evaluation methods for scientific reasoning, evidence synthesis, and hypothesis quality — where generic benchmarks fall short.
• Build evaluation tooling and reusable patterns that enable other teams to self‑serve.

Qualifications

Education:

Master's degree in AI/ML, Computer Science, Data Science, Computational Biology, Bioinformatics, Biomedical Engineering, Applied Mathematics, Biostatistics, or a related field required. PhD preferred.

Required

EXPERIENCE AND SKILLS:

We are looking for someone with 6+ years of hands‑on experience in AI/ML evaluation or data science (Master's) or 3+ years of industry experience (PhD). You should have experience designing and running evaluation frameworks, scientific benchmarks, or quality assessments for AI/ML systems, and hands‑on work with generative AI — large language models, retrieval‑augmented generation, agentic frameworks, and prompt engineering. We also value strong proficiency in Python and modern AI/ML tooling (evaluation harnesses, embedding models, vector databases, LLM APIs), the ability to translate expert scientific judgment into measurable criteria, rubrics, and reproducible protocols, and a collaborative, self‑driven approach to working across multidisciplinary teams.

Preferred

Other

English proficiency is required (written and verbal). This is a hybrid role based in Madrid or Barcelona, with limited travel (<10%), primarily within Europe.

Required Skills

Preferred Skills:

Advanced Analytics, Business Intelligence (BI), Coaching, Collaboration, Critical Thinking, Data Analysis, Database Management, Data Privacy Standards, Data Reporting, Data Savvy, Data Science, Data Visualization, Econometric Models, Process Improvements, Technical Credibility, Technologically Savvy, Workflow Analysis

The Anticipated Base Pay Range For This Position Is

€55,400.00 - €87,860.00

Benefits

In addition to base pay, we offer the following benefits*: an annual bonus with set target (% of pay) depending on pay grade / location, where the actual amount is based on the employees’ and companies’ performance of the previous calendar year, or sales commissions. Moreover, we offer vacation days, parental leave for a minimum of 12 weeks, bereavement leave, caregiver leave, volunteer leave, well‑being reimbursement, programs for financial, physical and mental health. We also offer service anniversary and recognition awards, and subject to the terms of their respective plans, employees - and in some location’s eligible dependents - can participate in several insurance plans. For more information, visit Employee benefits | Supporting well‑being & career growth | Johnson & Johnson Careers.

• This is for informative purposes only. Amounts and actual benefits may vary by location and are subject to change.