Quality Assurance Tester

Intellias; Portugal; Presencial; EUR 45 000 - 65 000; Tempo integral

Resumo da oferta

Intellias is seeking a QA / ML Tester – Evaluation Framework to ensure quality and reliability of AI evaluation systems across enterprise platforms. You will design validation strategies, test evaluator behavior across varied scenarios, and verify automated quality assessment pipelines in collaboration with AI engineers and platform teams.

The role sits in Intellias Core Architecture Team, contributing to a scalable, secure engineering platform that supports eCommerce, Digital Marketing, and IoT

Qualificações

• 4+ years QA or ML testing engineering.
• Experience with Python-based test automation using pytest.
• Experience validating evaluator correctness with known-good / known-bad sessions.
• Experience with on-demand mode integration testing in CI/CD.
• Experience validating online mode sampling accuracy.
• Knowledge of non-AgentCore runtime feasibility assessment methodology and AI/LLM system quality testing.

Responsabilidades

• Design, implement, and maintain automated test suites for AI evaluation frameworks and evaluation pipelines.
• Develop Python-based test automation using pytest to validate evaluator behavior, quality scoring, and framework reliability.
• Create and maintain known-good and known-bad test datasets, sessions, and workflows for evaluator correctness validation.
• Validate the accuracy and consistency of evaluation results across different agent workflows, prompts, tools, and execution scenarios.
• Design and execute integration tests for on-demand evaluation workflows integrated into CI/CD pipelines.
• Verify online evaluation behavior, sampling accuracy, and evaluation result consistency in production-like environments.
• Collaborate with AI and platform teams to identify edge cases, failure scenarios, and evaluation blind spots.
• Support feasibility assessments for applying evaluation frameworks to non-AgentCore runtimes and alternative AI execution environments.

Conhecimentos

QA testing

ML testing

Python

pytest

Evaluation framework testing

Automated testing

Ferramentas

CI/CD pipelines

OpenTelemetry

AWS

Descrição da oferta de emprego

We are looking for a QA / ML Tester – Evaluation Framework to ensure the quality, reliability, and correctness of evaluation systems used across enterprise AI and agent-based platforms. In this role, you will design and execute validation strategies, test evaluator behavior across a wide range of scenarios, and verify the accuracy of automated quality assessment pipelines. You will work closely with AI engineers, platform teams, and quality specialists to build confidence in evaluation results and support enterprise‑grade AI governance.

Our customer is a multinational corporation with more than a century of history and operations in over 180 countries. As part of its transformation, the company is driving the adoption of Reduced‑Risk Products (RRPs) for more than one billion consumers worldwide. Its technology landscape supports over 700 applications, requiring a scalable, secure, and highly reliable engineering platform.

At Intellias, you will join the Core Architecture Team, helping to engineer a comprehensive software ecosystem that powers best‑in‑class eCommerce, Digital Marketing, and IoT solutions. The Enterprise Platform provides engineering teams with shared services, technologies, and best practices that accelerate software delivery while ensuring quality, governance, compliance, and operational excellence across the software development lifecycle.

Requirements:

• 4+ years QA or ML testing engineering
• Python test automation (pytest)
• Evaluator correctness testing (known-good/known-bad session pairs)
• On-demand mode integration testing with CI/CD
• Online mode sampling accuracy validation
• Non-AgentCore runtime feasibility assessment methodology
• AI/LLM system quality testing

Nice-to-have

• AWS AgentCore Evaluation API testing
• OpenTelemetry trace‑based evaluation input testing
• Multi-evaluator execution correctness testing

Responsibilities:

• Design, implement, and maintain automated test suites for AI evaluation frameworks and evaluation pipelines.
• Develop Python-based test automation using pytest to validate evaluator behavior, quality scoring, and framework reliability.
• Create and maintain known-good and known-bad test datasets, sessions, and workflows for evaluator correctness validation.
• Validate the accuracy and consistency of evaluation results across different agent workflows, prompts, tools, and execution scenarios.
• Design and execute integration tests for on-demand evaluation workflows integrated into CI/CD pipelines.
• Verify online evaluation behavior, sampling accuracy, and evaluation result consistency in production‑like environments.
• Conduct functional testing of evaluation components, including evaluator execution flows, scoring logic, and result aggregation.
• Collaborate with AI and platform engineering teams to identify edge cases, failure scenarios, and evaluation blind spots.
• Validate workflow compliance, tool execution assessment, and end‑to‑end quality evaluation processes.
• Support feasibility assessments for applying evaluation frameworks to non‑AgentCore runtimes and alternative AI execution environments.
• Analyze defects, inconsistencies, and quality regressions within evaluation systems and provide actionable recommendations.
• Contribute to quality assurance standards, testing methodologies, and best practices for AI evaluation platforms.