Senior AI QA Engineer — Production ML Evaluation & Testing

Sii Poland; Województwo pomorskie; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Profit sharing

Medical care

Centre of internal trainings

Regular integration events

MySii app

Job summary

Sii Poland is seeking a Senior QA Engineer to help architect an AI-powered operating system for healthcare marketing. You will own quality, evaluate AI/ML systems, and design scalable evaluation pipelines across frontend Angular, Node.js backend, and Python microservices.

You will automate testing, monitor data drift, and drive performance strategies using Python and JavaScript/TypeScript, while integrating tests into CI/CD workflows and defining KPIs for model behavior.

Qualifications

• Minimum 5 years in software testing with at least 2 years on AI/ML apps or LLM workflows.
• Strong Python for backend/AI testing with solid JavaScript/TypeScript for Node.js/Angular testing.
• Experience with evaluation frameworks and selecting metrics for classification or generation-quality problems.
• Familiar with ML drift monitoring, observability tools, and production data monitoring.

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic harnesses.
• Develop automated scoring, assertions, and test suites for accuracy, reliability, hallucination, and tool-use.
• Simulate multi-step workflows to uncover edge cases in autonomous decision-making.
• Set up robust monitoring for data drift, concept drift, and model degradation in production.
• Implement telemetry and alerting for latency, token usage, and cost efficiency over time.
• Own end-to-end testing strategy across the stack: Angular frontend, Node.js backend, and Python AI services.
• Write scalable automated tests in Python and JavaScript/TypeScript.
• Perform API, integration, and performance/load testing for high-throughput microservices.
• Integrate AI evaluation into CI/CD pipelines for continuous quality gates.
• Define and track KPIs for software reliability and AI model behavior.

Skills

Python

JavaScript/TS

LLM testing

API testing

CI/CD testing

Datadog

Grafana

Test frameworks

Tools

LangSmith

Langfuse

EvidentlyAI

MLflow

Datadog

Grafana

Job description

Sii Poland is seeking a Senior QA Engineer to help architect an AI-powered operating system for healthcare marketing. You will own quality, evaluate AI/ML systems, and design scalable evaluation pipelines across frontend Angular, Node.js backend, and Python microservices.

You will automate testing, monitor data drift, and drive performance strategies using Python and JavaScript/TypeScript, while integrating tests into CI/CD workflows and defining KPIs for model behavior.