Quality Manager

Square One Resources; Barcelona; On-site; EUR 90,000 - 120,000; Full time

Job summary

Square One Resources is seeking a Quality Engineer Lead Expert to oversee QA for AI systems, including ML models, LLM-based applications, RAG pipelines and autonomous agents. The role covers functional, non-functional, security and compliance testing in an insurance context.

You will design test strategies for ML models and AI services, validate outputs, run adversarial testing and mentor junior testers while collaborating with AI engineers, data scientists, DevOps, and SMEs.

Qualifications

• Experience testing AI systems (ML models, LLMs, agents).
• Familiarity with data privacy and regulatory compliance in AI testing.
• Ability to design and execute test strategies for AI/ML pipelines.

Responsibilities

• Design and execute test strategies for ML models, LLMs and agents (accuracy, fairness, bias, explainability, robustness)
• Validate LLM and RAG outputs: grounding, hallucination, retrieval accuracy, vector stores, chunking logic
• Run adversarial, prompt-injection and red team testing
• Benchmark latency, throughput, concurrency and multi-agent performance; validate fallback and retry logic
• Build automated test frameworks for AI/ML pipelines, including synthetic data and regression testing of retrained models
• Define model quality KPIs, produce audit-ready evidence and mentor junior testers

Skills

AI testing

Python

API testing

Security testing

Performance testing

Cloud platforms

Big Data

Regulatory compliance

Tools

Python (test automation)

API testing tools

Cloud services

Big Data tooling

Job description

We are looking for a Quality Engineer Lead Expert to lead quality assurance of our AI systems: ML models, LLM-based applications, RAG pipelines and autonomous agents. The role covers functional, non-functional, security and compliance testing in an insurance context. It works closely with AI engineers, data scientists, DevOps teams and SMEs.

Main Responsibilities

• Design and execute test strategies for ML models, LLMs and agents (accuracy, fairness, bias, explainability, robustness)
• Validate LLM and RAG outputs: grounding, hallucination, retrieval accuracy, vector stores, chunking logic
• Run adversarial, prompt-injection and red team testing
• Benchmark latency, throughput, concurrency and multi-agent performance; validate fallback and retry logic
• Build automated test frameworks for AI/ML pipelines, including synthetic data and regression testing of retrained models
• Define model quality KPIs, produce audit-ready evidence and mentor junior testers

Experience Required

• Software testing focused on AI systems (ML models, LLMs, agents)
• Python for test automation, data validation and AI testing scripts
• API, performance and security testing of AI services and model endpoints
• AI ethics, data privacy and regulatory compliance testing
• Testing AI on cloud platforms and Big Data ecosystems at scale