Senior AI QA Engineer - Build & Monitor ML Systems

Sii Poland; Toruń; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Great Place to Work

Solid financial situation

Contracts with the biggest brands

Centre of internal trainings

Many experts you can learn from

Open and accessible management team

Profit sharing

Passion Sponsorship program

Job summary

Sii Poland invites a Senior QA Engineer to join our team and help build an AI-powered operating system for healthcare marketing. You will own quality and evaluation for complex AI systems, design scalable evaluation pipelines, and ensure robust monitoring across the stack.

You will work with Python, JavaScript/TypeScript, and CI/CD processes to secure production-grade testing, API and performance validation, and data/query telemetry. Advanced English is required.

Qualifications

• 5+ years in software testing or QA, with 2+ years in AI/ML testing or related workflows
• Proficiency in Python for backend/AI testing scripts; JavaScript/TypeScript for Node.js/Angular testing
• Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas or MLflow
• Familiarity with ML drift monitoring concepts and observability tools (Datadog, Grafana)
• Strong system-level design skills to build prod-grade test and evaluation infra
• Advanced level of English

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic systems
• Develop automated scoring, assertions, and test suites for accuracy, reliability, and tool-use
• Simulate complex agent workflows to uncover edge cases in autonomous decision-making
• Set up monitoring for data drift and model performance in production
• Implement telemetry and alerting for latency, token usage, and cost efficiency
• Own end-to-end testing across frontend, backend, and AI microservices
• Write clean, scalable test scripts in Python and JavaScript/TypeScript
• Perform API, integration, and performance testing for microservices
• Integrate AI testing into CI/CD pipelines and gate quality before deployment
• Define and track KPIs for software reliability and AI model behavior

Skills

Python testing

JavaScript/TypeScript

LLM testing

ML monitoring

Observability tools

System design

Advanced English

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Datadog

Grafana

Job description

Sii Poland invites a Senior QA Engineer to join our team and help build an AI-powered operating system for healthcare marketing. You will own quality and evaluation for complex AI systems, design scalable evaluation pipelines, and ensure robust monitoring across the stack.

You will work with Python, JavaScript/TypeScript, and CI/CD processes to secure production-grade testing, API and performance validation, and data/query telemetry. Advanced English is required.