Senior AI QA Engineer: Elevate ML Testing & Evaluation

Sii Poland; Łódź; On-site; PLN 180,000 - 260,000; Full time

Benefits offered by this job

Great Place to Work

Profit sharing

Centre of internal trainings

Medical care

Open management

Regular team events

Job summary

Sii Poland is seeking a Senior QA Engineer to shape our AI-powered healthcare marketing platform. You will own end-to-end testing across the stack—Angular frontend, Node.js backend, and Python AI microservices—designing robust evaluation pipelines and monitoring for drift and performance.

Applicants should have 5+ years in software testing with 2+ years in AI/ML contexts, strong Python and JavaScript skills, and experience with AI evaluation tools and observability platforms.

Qualifications

• 5+ years in software testing, with 2+ years in AI/ML testing or LLM workflows
• Python for backend/AI testing; JavaScript/TypeScript for Node.js and Angular testing
• Experience with AI evaluation frameworks and selecting metrics for classification or generation-quality
• Familiarity with ML drift monitoring and observability tools such as Datadog or Grafana
• Strong system-level design skills to build production-grade test/evaluation infra
• Advanced level of English

Responsibilities

• Designing evaluating frameworks for LLMs and agentic harnesses
• Develop automated scoring, assertions, and test suites for AI system reliability
• Simulate complex agent workflows to reveal edge cases in autonomous decisions
• Collaborate with teams to set up monitoring for data drift and model performance in production
• Implement telemetry and alerting for latency, token usage, and output quality
• Own end-to-end testing strategy across Angular frontend, Node.js backend, and Python AI microservices
• Write clean, scalable test scripts in Python and JavaScript/TypeScript
• Perform API, integration, and performance/load testing for microservices
• Integrate AI evaluation into CI/CD pipelines for gating quality before deployment
• Define and track KPIs for software reliability and AI behavior

Skills

Python scripting

JavaScript/TypeScript testing

English proficiency

AI/ML testing experience

Observability basics

CI/CD integration

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Datadog

Grafana

Angular

Node.js

Job description

Sii Poland is seeking a Senior QA Engineer to shape our AI-powered healthcare marketing platform. You will own end-to-end testing across the stack—Angular frontend, Node.js backend, and Python AI microservices—designing robust evaluation pipelines and monitoring for drift and performance.

Applicants should have 5+ years in software testing with 2+ years in AI/ML contexts, strong Python and JavaScript skills, and experience with AI evaluation tools and observability platforms.