Senior AI QA Engineer: Build & QA AI Evaluation Pipelines

Sii Poland; Wrocław; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Great Place to Work

Solid financial situation

Centre of internal trainings

Profit sharing

Medical care

Job summary

Sii Poland in Wroclaw is seeking a Senior QA Engineer to help build an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems across the full stack, ensuring stability of the architecture while scaling AI evaluation pipelines.

You will design and implement automated test suites using Python and JavaScript/TypeScript, work on CI/CD integration, and collaborate with teams to monitor data drift, model performance, and latency.

Qualifications

• 5+ years in software testing or quality engineering, with 2+ years in AI/ML applications.
• Proficiency in Python and JavaScript/TypeScript for testing.
• Experience with evaluation frameworks LangSmith, Langfuse, EvidentlyAI, Ragas, MLflow.
• Familiarity with ML drift monitoring and observability tools like Datadog or Grafana.
• Strong system-level design for production-grade test infrastructure.
• Excellent English communication.

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic harnesses.
• Develop automated scoring mechanisms and test suites for agent accuracy and reliability.
• Simulate complex agent workflows to uncover edge cases in autonomous decision-making.
• Collaborate with engineering to set up monitoring for data drift and model performance.
• Implement telemetry and alerting to track latency, token usage, and cost efficiency.
• Own end-to-end testing across Angular frontend, Node.js backend, and Python AI microservices.
• Write clean, scalable test scripts in Python and JavaScript/TypeScript.
• Perform API, integration, and performance/load testing for high-throughput services.
• Integrate AI evaluation harnesses into CI/CD pipelines for quality gates.
• Define and track KPIs for software reliability and AI model behavior.

Skills

Python

JavaScript/TypeScript

Test automation

AI/ML testing

CI/CD integration

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Datadog

Grafana

Job description

Sii Poland in Wroclaw is seeking a Senior QA Engineer to help build an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems across the full stack, ensuring stability of the architecture while scaling AI evaluation pipelines.

You will design and implement automated test suites using Python and JavaScript/TypeScript, work on CI/CD integration, and collaborate with teams to monitor data drift, model performance, and latency.