Senior AI QA Engineer: LLM Testing & Drift Monitoring

Sii Poland; Białystok; On-site; PLN 180,000 - 270,000; Full time

Benefits offered by this job

Great Place to Work

Centre of internal trainings

Profit sharing

MySii app

Medical care

Job summary

Sii Poland is seeking a Senior QA Engineer to join our team and build a robust AI-powered operating system for healthcare marketing. You will own QA, evaluation, and monitoring for complex AI systems and drive end-to-end testing across full-stack services including Angular, Node.js, and Python microservices.

The role emphasizes developing scalable test frameworks for AI/ML, implementing telemetry, and establishing KPIs to ensure reliability and quality of AI models in production.

Qualifications

• 5+ years of experience in software testing or quality engineering
• 2+ years focused on testing or evaluating AI/ML applications, LLM workflows, or agentic systems
• Proficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems
• Experience with evaluation frameworks and metrics selection for classification or generation-quality problems
• Familiarity with ML drift monitoring concepts and observability tools
• Strong English language skills

Responsibilities

• Designing, building, and maintaining evaluation frameworks for LLMs and agentic harnesses
• Developing automated scoring mechanisms, assertions, and test suites to measure agent accuracy, reasoning reliability, and tool-use correctness
• Simulating complex multi-step agent workflows to uncover edge cases in autonomous decision-making
• Setting up monitoring frameworks for data drift and model performance degradation in production
• Implementing telemetry and alerting to track latency, token usage, and output quality
• Owning end-to-end testing strategy across the stack (Angular frontend, Node.js backend, Python AI services)
• Writing clean, scalable test scripts in Python and JavaScript/TypeScript
• Performing API, integration, and performance/load testing for high-throughput microservices
• Integrating AI evaluation into CI/CD pipelines for quality gate enforcement
• Defining KPIs for software reliability and AI model behavior

Skills

Python

JavaScript/TypeScript

LLM evaluation

AI/ML testing

English proficiency

CI/CD pipelines

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Job description

Sii Poland is seeking a Senior QA Engineer to join our team and build a robust AI-powered operating system for healthcare marketing. You will own QA, evaluation, and monitoring for complex AI systems and drive end-to-end testing across full-stack services including Angular, Node.js, and Python microservices.

The role emphasizes developing scalable test frameworks for AI/ML, implementing telemetry, and establishing KPIs to ensure reliability and quality of AI models in production.