Senior Engineer – GenAI Quality Assurance

Johnson Johnson; Cornellà de Llobregat; On-site; EUR 70,000 - 90,000; Full time

Job summary

Jobtailor in Cataluña is seeking a QA engineer focused on GenAI to design and implement automated and manual QA strategies for GenAI systems, with strong Python skills and experience with LLM-based testing.

You will build regression tests, generate synthetic data, validate prompts and model behavior, execute tests, debug pipelines, and support deployment while leading design reviews and coordinating with data scientists and developers.

Qualifications

• 3+ years of software engineering experience in production systems.
• Experience with GenAI systems and RAG/graph frameworks.
• Strong Python and API testing, data validation, and pipelines.
• Familiarity with LLM APIs and cloud platforms (AWS/GCP/Azure).

Responsibilities

• Design and implement QA strategies for GenAI systems.
• Build regression tests and synthetic data workflows.
• Validate prompt logic, model behavior and metrics.
• Execute tests, debug pipelines, and support deployment.
• Lead design reviews and establish CI/CD testing standards.
• Collaborate with data scientists and developers; conduct code reviews.

Skills

Python

LLM-based systems

RAG frameworks

APIs testing

Data validation

CI/CD testing

Tools

Git

AWS Bedrock

Azure Foundry

GCP

Test harnesses

Job description

Responsibilities

• Design and implement automated and manual QA strategies for GenAI systems
• Build regression tests, generate synthetic test data, build test harnesses and evaluation pipelines for accuracy
• Validate prompt logic, model behavior and evaluation metrics
• Execute tests and debug pipelines
• Support deployment of GenAI products
• Lead system design reviews
• Collaborate closely with data scientists and developers
• Establish testing standards and workflows for continuous integration and deployment
• Conduct code reviews and document system design decisions

Requirements

• 3+ years of experience as a software engineer
• Familiarity with RAG & Graph frameworks
• 5+ years of experience in production systems
• 1+ year experience in GenAI systems
• Advanced Python skills
• Experience with LLM-based systems
• Hands‑on experience with: LLM APIs, AWS Bedrock, Azure Foundry, GCP
• Proven ability to implement test frameworks
• Knowledge of API testing, data validation, data pipeline testing
• Experience with version control (Git)

Core Competencies

Demonstrates expertise in designing and implementing QA strategies for GenAI systems, with advanced skills in Python and experience in LLM-based systems. Proficient in establishing testing standards, conducting code reviews, and collaborating with cross‑functional teams to ensure product quality.