Senior Engineer – GenAI Quality Assurance

Johnson Johnson; Madrid; On-site; EUR 55,000 - 90,000; Full time

Job summary

Jobtailor seeks a seasoned software engineer/QA specialist to design and execute automated and manual QA strategies for GenAI systems in Madrid. You will build regression tests, synthetic data, and evaluation pipelines to verify accuracy, grounding, robustness, and safety.

You will validate prompt logic and model behavior, run tests, review code, and contribute to deployment of GenAI products. Collaboration with data scientists and developers will be essential to embed testing checkpoints and

Qualifications

• Strong software engineering background with QA ownership.
• 3+ years as a software engineer or QA in production systems.
• Advanced Python skills and experience with GenAI systems.
• Proven ability to implement test frameworks for GenAI applications.
• Experience with API testing, data validation, and data pipelines.

Responsibilities

• Design and implement automated and manual QA strategies for GenAI systems.
• Build regression tests, synthetic data, and evaluation pipelines for accuracy, grounding, robustness, and safety.
• Validate prompt logic and model behavior; execute tests and debug pipelines.
• Support deployment of GenAI products and lead design reviews for reliability.
• Collaborate with data scientists and developers to embed testing checkpoints and CI/CD workflows.

Skills

Python

GenAI QA

LLM testing

Test framework design

Data validation

Problem solving

Education

Bachelor's degree in CS/Engineering

Tools

LLM APIs

Git

Cloud platforms (AWS, Azure, GCP)

CI/CD

Job description

Responsibilities

• Design and implement automated and manual QA strategies for GenAI systems
• Build regression tests, generate synthetic test data, build test harnesses and evaluation pipelines for accuracy, grounding, robustness, safety
• Validate prompt logic, model behavior and evaluation metrics
• Execute tests and debug pipelines
• Support deployment of GenAI products
• Lead system design reviews, advocating best practices for architecture and reliability of GenAI applications
• Collaborate closely with data scientists and developers to embed testing and quality checkpoints
• Establish testing standards and workflows for continuous integration and deployment
• Conduct code reviews with a focus on design, testability, and maintainability
• Document system design decisions, test cases, and best practices
• Close collaboration with domain experts and stakeholders to validate developed solutions

Requirements

• Strong software engineering background with QA ownership
• Familiarity with RAG & Graph frameworks
• 3+ years of experience as a software engineer (preferably 5), or QA engineer in production systems, preferably including at least 1 year experience in GenAI systems
• Advanced Python skills
• Experience working with LLM-based systems in real environments
• Hands-on experience with: LLM APIs, AWS Bedrock (Anthropic & OSS), Azure Foundry (OpenAI), GCP (Gemini), Cloud platforms (AWS, Azure, GCP)
• Proven ability to implement test frameworks for GenAI applications
• Knowledge of API testing, data validation, and data pipeline testing
• Experience with version control (Git)
• Experience with building and deploying AI/ML products through the full product lifecycle
• Good understanding of scientific computing and ML frameworks
• Solid understanding of safety, bias, and failure modes of GenAI systems
• Comfortable working in a relatively fast-moving environment where priorities may shift and initiative is valued
• Proactive problem-solver who can anticipate needs and find solutions
• Comfortable working independently, but also a strong team player.