Senior AI QA Engineer — AI/ML Testing & Evaluation

Sii Poland; Warszawa; On-site; PLN 200,000 - 280,000; Full time

Benefits offered by this job

Great Place to Work

Centre of internal trainings

Profit sharing

Medical care

Open and accessible management team

Job summary

Sii Poland seeks a Senior QA Engineer to join our team and contribute to building an AI-powered operating system for healthcare marketing, owning quality, evaluation, and monitoring for complex AI systems.

You will design evaluation pipelines, monitor data drift and model performance in production, and integrate testing into CI/CD across Angular frontend, Node.js backend, and Python AI microservices. This role emphasizes scalable automation, telemetry, and measurable KPIs for reliability.

Qualifications

• 5+ years in software testing or quality engineering, with 2+ years focused on AI/ML testing or evaluation
• Proficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems
• Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow, and ability to select metrics for classification or generation-quality problems
• Familiarity with ML drift monitoring concepts, metrics tracking, and observability tools like Datadog or Grafana
• Strong system-level design skills and ability to build production-grade test and evaluation infrastructure from the ground up
• Advanced level of English

Responsibilities

• Designing, building, and maintaining evaluation frameworks for LLMs and custom agentic harnesses
• Developing automated scoring mechanisms, assertions, and test suites to measure agent accuracy, reasoning reliability, hallucination rates, and tool-use correctness
• Simulating complex, multi-step agent workflows to uncover edge cases in autonomous decision-making
• Collaborating with engineering teams to set up robust monitoring frameworks for data drift, concept drift, and model performance degradation in production
• Implementing telemetry and alerting systems to track latency, token usage, cost efficiency, and output quality over time
• Owning the end-to-end testing strategy across our stack, ensuring seamless integration between Angular frontend, Node.js backend services, and Python-based AI microservices
• Writing clean, maintainable, and scalable automated test scripts in Python and JavaScript/TypeScript
• Performing rigorous API testing, integration testing, and performance/load testing for high-throughput microservices
• Integrating AI evaluation and testing harnesses into our CI/CD pipelines to ensure continuous quality gate enforcement before deployment
• Defining and tracking KPIs for software reliability and AI model behavior

Skills

Python

JavaScript/TypeScript

English

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Datadog

Grafana

Angular

Node.js

Job description

Sii Poland seeks a Senior QA Engineer to join our team and contribute to building an AI-powered operating system for healthcare marketing, owning quality, evaluation, and monitoring for complex AI systems.

You will design evaluation pipelines, monitor data drift and model performance in production, and integrate testing into CI/CD across Angular frontend, Node.js backend, and Python AI microservices. This role emphasizes scalable automation, telemetry, and measurable KPIs for reliability.