Postdoctoral Scholar, AI Evaluation & Standards

7300-Janssen-Cilag S.A. Legal Entity; Madrid; Presencial; EUR 43.600 - 70.150; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Annual bonus

Vacation days

Parental leave

Well-being reimbursement

Descripción de la vacante

Johnson & Johnson, a leader in healthcare, seeks a Post Doctoral Researcher in Data Analytics & Computational Sciences to design evaluation frameworks for GenAI tools used in pharmaceutical R&D. You will build datasets, run expert reviews with cross-functional teams, and define release criteria as systems move from prototype to broader use.

Required are a PhD or equivalent in biomedical science or related field, strong Python and data science skills, and experience translating expert judgment

Formación

• PhD or equivalent research experience in biomedical science, computational biology, bioinformatics, AI/ML, data science, clinical research, regulatory science, biostatistics, pharmaceutical sciences, or related field.
• Experience translating expert judgment into criteria, rubrics, datasets, protocols, or measurable outcomes.
• Experience designing or applying evaluation methods, benchmark datasets, annotation protocols, validation studies, quality reviews, or assessment frameworks.
• Interest in testing GenAI systems, including LLMs, RAG, and AI agents.
• Proficiency in Python and common data science or machine learning tools.

Responsabilidades

• Design evaluation frameworks, rubrics, and criteria for GenAI tools used across pharmaceutical R&D.
• Develop therapeutics-area-specific criteria with business and scientific teams.
• Build benchmark datasets, reference answer sets, annotation guides, and evaluation datasets.
• Run expert reviews with scientific, clinical, regulatory, medical, data science, and engineering teams.
• Test LLM, RAG, and agent performance including accuracy, source grounding, and citation fidelity.
• Analyze failure patterns and translate findings into improvements in prompts and retrieval methods.
• Define release criteria for systems moving from prototype to limited release or expansion.
• Review emerging evaluation methods and adapt useful approaches.
• Document methods, findings, and recommendations for consistent evaluation practices.
• Design agentic judge methods to evaluate GenAI outputs and support expert reviews.

Conocimientos

Python

Data science

Benchmarking

Rubric design

Biomedical research

Technical communication

Educación

PhD or equivalent

Herramientas

LLM APIs

Embeddings

Vector databases

Descripción del empleo

At Johnson & Johnson,we believe health is everything. Our strength in healthcare innovation empowers us to build aworld where complex diseases are prevented, treated, and cured,where treatments are smarter and less invasive, andsolutions are personal.Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity.Learn more at jnj.com. As guided by Our Credo, Johnson & Johnson is responsible to our employees who work with us throughout the world. We provide an inclusive work environment where each person is considered as an individual. At Johnson & Johnson, we respect the diversity and dignity of our employees and recognize their merit.

Job Function: Career Programs

Job Sub Function: Post Doc – Data Analytics & Computational Sciences

Job Category: Career Program

All Job Posting Locations:

• Barcelona, Spain
• Beerse, Antwerp, Belgium
• Madrid, Spain
• Raritan, New Jersey, United States of America
• Titusville, New Jersey, United States of America

Job Description: Job description At J&J we are developing Generative AI solutions to support pharmaceutical R&D, including literature review, evidence synthesis, document Q&A, therapeutic area knowledge search, translational science workflows, and R&D decision support. These systems need to be tested before teams use them in scientific workflows. In pharmaceutical R&D, a useful AI response depends on the question, user, source material, therapeutic area, and risk of error. We are looking for a postdoctoral researcher to help design methods that test whether GenAI tools produce answers that are accurate, evidence-grounded, traceable, usable, and appropriate for the intended task. The role reports to the Associate Director, Generative AI Evaluation & Quality Standards. The team defines how J&J Innovative Medicine evaluates GenAI systems before use and helps determine when they are ready for release, expansion, or improvement.

Key responsibilities

• Design evaluation frameworks, rubrics, and criteria for GenAI tools used across pharmaceutical R&D.
• Develop therapeutic-area-specific criteria with business and scientific teams to reflect domain and use-case quality needs.
• Build benchmark datasets, reference answer sets, annotation guides, and evaluation datasets.
• Run expert reviews with scientific, clinical, regulatory, medical, data science, and engineering teams.
• Test LLM, RAG, and agent performance, including accuracy, source grounding, retrieval quality, citation fidelity, task completion, robustness, safety, and usability.
• Analyze failure patterns such as unsupported claims, incorrect reasoning, poor evidence use, missing uncertainty, weak traceability, or failure to follow instructions.
• Translate evaluation findings into improvements in prompts, retrieval methods, agent workflows, tools, and user experience.
• Help define release criteria for systems moving from prototype to limited release, expanded use, or product support.
• Review emerging evaluation methods and adapt useful approaches for pharmaceutical R&D.
• Document methods, findings, and recommendations so teams can apply consistent evaluation practices.
• Design and develop agentic judge methods to evaluate GenAI outputs against defined criteria, flag evidence gaps or unsupported claims, and support expert review workflows.

Qualifications

• Education PhD or equivalent research experience in biomedical science, computational biology, bioinformatics, AI/ML, data science, clinical research, regulatory science, biostatistics, pharmaceutical sciences, or a related field.
• Experience and skills Required Understanding of biomedical science, pharmaceutical R&D, therapeutic area science, translational science, clinical development, regulatory science, biomedical informatics, data science, or related areas.
• Experience translating expert judgment into criteria, rubrics, datasets, protocols, or measurable outcomes.
• Experience designing or applying evaluation methods, benchmark datasets, annotation protocols, validation studies, quality reviews, or assessment frameworks.
• Interest in testing GenAI systems, including LLMs, RAG, and AI agents.
• Proficiency in Python and common data science or machine learning tools.
• Ability to analyze model outputs, compare performance, identify failure patterns, and recommend improvements.
• Clear written and verbal communication skills.
• Preferred Experience with LLM APIs, embeddings, vector databases, prompt engineering, agent frameworks, or AI evaluation tools.
• Experience evaluating retrieval quality, generated answers, multi-step workflows, tool use, scientific reasoning, citation quality, or evidence-grounded outputs.
• Experience designing expert review workflows, annotation instructions, adjudication processes, or inter-rater reliability analyses.
• Domain knowledge in one or more biomedical or therapeutic areas.
• Familiarity with biomedical data standards, structured scientific or clinical data, ontologies, knowledge graphs, CDISC, FHIR, or related frameworks.
• Publications or applied research in AI evaluation, NLP, biomedical informatics, machine learning, data science, computational biology, bioinformatics, or a related field.
• Required Skills: GenAI evaluation, Python, data science, benchmarking, rubric design, biomedical research, technical communication.
• Preferred Skills: RAG evaluation, agent evaluation, biomedical informatics, expert review, annotation protocols, therapeutic area expertise, responsible AI.
• Required Skills: Preferred Skills:

The anticipated base pay range for this position is: €43,600.00 - €70,150.00

Benefits

• an annual bonus with set target (% of pay) depending on pay grade / location, where the actual amount is based on the employees’ and companies’ performance of the previous calendar year, or sales commissions.
• vacation days, parental leave for a minimum of 12 weeks, bereavement leave, caregiver leave, volunteer leave, well-being reimbursement, programs for financial, physical and mental health.
• service anniversary and recognition awards, and subject to the terms of their respective plans, employees - and in some location’s eligible dependents - can participate in several insurance plans.

At Johnson & Johnson,we believe health is everything. Our strength in healthcare innovation empowers us to build aworld where complex diseases are prevented, treated, and cured,where treatments are smarter and less invasive, andsolutions are personal.Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity.Learn more at https://www.jnj.com/.

Do Not Sell or Share My Personal Information

Limit the Use of My Personal Information