AI Evaluation QA Engineer — ML Testing & Validation

Intellias; Town of Poland (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Intellia is seeking a QA / ML Tester — Evaluation Framework to ensure the quality, reliability, and correctness of evaluation systems used across enterprise AI and agent-based platforms. You will design and execute validation strategies for evaluation frameworks, test evaluator behavior across multiple scenarios, and verify the accuracy of automated quality assessment pipelines.

Collaborate with AI engineers, platform teams, and quality specialists to establish confidence in evaluation results

Qualifications

• Proven experience building and executing Python-based test suites for ML/AI systems.
• Experience validating evaluation pipelines and scoring logic.
• Familiarity with CI/CD workflows and on-demand vs online evaluation modes.

Responsibilities

• Design, implement, and maintain automated test suites for AI evaluation frameworks and pipelines.
• Develop Python-based test automation using pytest to validate evaluator behavior and framework reliability.
• Create known-good and known-bad datasets, sessions, and workflows for evaluator correctness validation.
• Validate accuracy and consistency of evaluation results across agent workflows, prompts, tools, and execution scenarios.
• Design and execute integration tests for on-demand evaluation workflows in CI/CD pipelines.
• Verify online evaluation behavior and sampling accuracy in production-like environments.
• Conduct functional testing of evaluation components including evaluator execution flows and scoring logic.
• Collaborate with AI and platform teams to identify edge cases and evaluation blind spots.

Skills

Python test automation

Evaluator correctness testing

CI/CD integration testing

Online sampling validation

Non-AgentCore feasibility assessment

Tools

pytest

Job description

Intellia is seeking a QA / ML Tester — Evaluation Framework to ensure the quality, reliability, and correctness of evaluation systems used across enterprise AI and agent-based platforms. You will design and execute validation strategies for evaluation frameworks, test evaluator behavior across multiple scenarios, and verify the accuracy of automated quality assessment pipelines.

Collaborate with AI engineers, platform teams, and quality specialists to establish confidence in evaluation results