Senior QA Engineer - AI Evaluation & Validation

Intellias; Portugal; Presencial; EUR 45 000 - 65 000; Tempo integral

Resumo da oferta

Intellias is seeking a QA / ML Tester – Evaluation Framework to ensure quality and reliability of AI evaluation systems across enterprise platforms. You will design validation strategies, test evaluator behavior across varied scenarios, and verify automated quality assessment pipelines in collaboration with AI engineers and platform teams.

The role sits in Intellias Core Architecture Team, contributing to a scalable, secure engineering platform that supports eCommerce, Digital Marketing, and IoT

Qualificações

• 4+ years QA or ML testing engineering.
• Experience with Python-based test automation using pytest.
• Experience validating evaluator correctness with known-good / known-bad sessions.
• Experience with on-demand mode integration testing in CI/CD.
• Experience validating online mode sampling accuracy.
• Knowledge of non-AgentCore runtime feasibility assessment methodology and AI/LLM system quality testing.

Responsabilidades

• Design, implement, and maintain automated test suites for AI evaluation frameworks and evaluation pipelines.
• Develop Python-based test automation using pytest to validate evaluator behavior, quality scoring, and framework reliability.
• Create and maintain known-good and known-bad test datasets, sessions, and workflows for evaluator correctness validation.
• Validate the accuracy and consistency of evaluation results across different agent workflows, prompts, tools, and execution scenarios.
• Design and execute integration tests for on-demand evaluation workflows integrated into CI/CD pipelines.
• Verify online evaluation behavior, sampling accuracy, and evaluation result consistency in production-like environments.
• Collaborate with AI and platform teams to identify edge cases, failure scenarios, and evaluation blind spots.
• Support feasibility assessments for applying evaluation frameworks to non-AgentCore runtimes and alternative AI execution environments.

Conhecimentos

QA testing

ML testing

Python

pytest

Evaluation framework testing

Automated testing

Ferramentas

CI/CD pipelines

OpenTelemetry

AWS

Descrição da oferta de emprego

Intellias is seeking a QA / ML Tester – Evaluation Framework to ensure quality and reliability of AI evaluation systems across enterprise platforms. You will design validation strategies, test evaluator behavior across varied scenarios, and verify automated quality assessment pipelines in collaboration with AI engineers and platform teams.

The role sits in Intellias Core Architecture Team, contributing to a scalable, secure engineering platform that supports eCommerce, Digital Marketing, and IoT