Senior AI QA Engineer - Automation & ML Evaluation

Sii Poland; Poznań; On-site; PLN 180,000 - 280,000; Full time

Job summary

Sii Poland is seeking a Senior QA Engineer to join our team and help build an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for AI systems, ensuring stability of our full-stack architecture while scaling AI evaluation pipelines and drift monitoring.

You will design evaluation frameworks, implement automated tests in Python and JS/TS, and lead testing across Angular frontend, Node.js backend, and Python microservices.

Qualifications

• Minimum 5 years of experience in software testing or quality engineering.
• At least 2 years focused on testing or evaluating AI/ML applications, LLM workflows, or agentic systems.
• Proficiency in Python for backend AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems.
• Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow, and the ability to select appropriate metrics for classification or generation-quality problems.

Responsibilities

• Designing, building, and maintaining evaluation frameworks for Large Language Models (LLMs) and custom agentic harnesses.
• Developing automated scoring mechanisms, assertions, and test suites to measure agent accuracy, reasoning reliability, hallucination rates, and tool-use correctness.
• Simulating complex, multi-step agent workflows to uncover edge cases in autonomous decision-making.
• Collaborating with engineering teams to set up robust monitoring frameworks for data drift, concept drift, and model performance degradation in production.
• Implementing telemetry and alerting systems to track latency, token usage, cost efficiency, and output quality over time.
• Owning the end-to-end testing strategy across our entire stack, ensuring seamless integration between the Angular frontend, Node.js backend services, and Python-based AI microservices.
• Writing clean, maintainable, and scalable automated test scripts in Python and JavaScript/TypeScript.
• Performing rigorous API testing, integration testing, and performance/load testing for high-throughput microservices.
• Integrating AI evaluation and testing harnesses into our CI/CD pipelines to ensure continuous quality gate enforcement before deployment.
• Defining and tracking key quality metrics (KPIs) for software reliability and AI model behavior.

Skills

System design

Python

English proficiency

Test automation

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Job description

Sii Poland is seeking a Senior QA Engineer to join our team and help build an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for AI systems, ensuring stability of our full-stack architecture while scaling AI evaluation pipelines and drift monitoring.

You will design evaluation frameworks, implement automated tests in Python and JS/TS, and lead testing across Angular frontend, Node.js backend, and Python microservices.