AI Automation QA

ITDS; Portugal; Híbrido; EUR 60 000 - 90 000; Tempo integral

Resumo da oferta

ITDS is seeking a Senior AI & Automation QA Engineer to build and maintain testing infrastructure for AI features across conversational AI, productivity tools, and rating systems. You will ensure accuracy, safety, and regression-free performance in production-grade AI applications.

Based in Porto with remote work options, the role emphasizes AI-specific testing challenges, dataset design, and integration into CI/CD pipelines for robust evaluation and observability of model outputs.

Qualificações

• 3+ years of QA/SDET experience, focusing on automation testing.
• Hands-on experience building automated test frameworks with Playwright, Cypress, or Selenium.
• Experience addressing AI-specific testing challenges, including prompt regression and bias detection.

Responsabilidades

• Develop and maintain automated test frameworks to validate AI features.
• Design evaluation datasets and automated pipelines for AI testing.
• Analyze AI outputs to identify errors, hallucinations, and biases for continuous improvement.
• Integrate AI evaluation and functional tests into CI/CD pipelines.

Conhecimentos

Automation Testing

Root Cause Analysis

Ferramentas

Playwright

Cypress

Selenium

Descrição da oferta de emprego

Unleash innovation in AI testing — drive the future of safe, reliable AI-powered solutions!

Porto-based opportunity with remote work

As a Senior AI & Automation QA Engineer, you will be working for our client, a leader in AI-powered features across conversational AI, productivity tools, and rating systems. You will play a pivotal role in building and maintaining the testing infrastructure that ensures the accuracy, safety, and regression-free performance of cutting-edge AI applications. This position offers a dynamic environment dedicated to advancing AI reliability and fostering professional growth.

Your main responsibilities:

• Develop and maintain automated test frameworks using tools like Playwright, Cypress, or Selenium to ensure AI features perform flawlessly.
• Design and implement evaluation datasets and automated pipelines, including innovative LLM-based judgment approaches.
• Analyze AI outputs to identify hallucinations, errors, and biases, supporting continuous improvement.
• Integrate AI evaluation and functional tests into CI/CD pipelines for seamless deployment.
• Utilize AI observability and tracing tools for debugging model behavior in production environments.
• Craft and execute test cases for conversational AI, including dialogue flows, edge cases, and adversarial prompts.
• Collaborate on ensuring AI outputs meet fairness, bias, and compliance standards through rigorous testing.

You're ideal for this role if you have:

• 3+ years of experience in QA/SDET roles, especially in automation testing.
• Hands-on experience building automated test frameworks with Playwright, Cypress, Selenium, or similar.
• Understanding of AI-specific testing challenges, including prompt regression, hallucination detection, and output scoring.
• Experience designing evaluation datasets and using AI observability/tracing tools.
• Knowledge of fairness, bias mitigation, and compliance testing for AI models.
• Familiarity with CI/CD pipelines to automate testing processes.
• Strong root-cause analysis skills covering both code and probabilistic AI outputs.

It is a strong plus if you have:

• Experience with AI-assisted test generation or self-healing scripts.
• Knowledge of LLM evaluation approaches and AI debugging tools.
• Certifications related to QA or AI safety.

Language Required for the role: Fluent English

Eligibility to work in Europe: Only candidates with an existing legal right to work in the European Union will be considered for this role.