Senior AI QA Engineer: ML Evaluation & Drift Monitoring

Sii Poland; Piła; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Great Place to Work

Profit sharing

Center of internal trainings

Medical care

Job summary

Sii Poland is seeking a Senior QA Engineer to contribute to an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems across the full stack, from frontend Angular to Python microservices, ensuring stability and scalable testing frameworks.

Join a team building production-grade AI solutions, designing evaluation pipelines, drift monitoring, and robust test automation in Python and JavaScript/TypeScript.

Qualifications

• 5+ years in software testing or quality engineering with AI/ML focus
• 2+ years testing AI/ML applications, LLM workflows or agentic systems
• Proficiency in Python for backend/AI testing scripts
• Experience with Node.js/Angular testing ecosystems

Responsibilities

• Design and maintain evaluation frameworks for LLMs
• Develop automated scoring and test suites for AI accuracy
• Simulate complex agent workflows to reveal edge cases
• Set up monitoring for data drift and model performance
• Implement telemetry and alerting to track latency and cost
• Lead end-to-end testing across frontend, backend and AI services
• Write test scripts in Python and JavaScript/TypeScript
• Perform API and performance testing for high-throughput services
• Integrate AI evaluation into CI/CD pipelines
• Define KPIs for software reliability and AI behavior

Skills

Python testing

JS/TS testing

AI/ML testing

English (Advanced)

Tools

LangSmith

Langfuse

EvidentlyAI

MLflow

Datadog

Grafana

Job description

Sii Poland is seeking a Senior QA Engineer to contribute to an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems across the full stack, from frontend Angular to Python microservices, ensuring stability and scalable testing frameworks.

Join a team building production-grade AI solutions, designing evaluation pipelines, drift monitoring, and robust test automation in Python and JavaScript/TypeScript.