Senior AI QA Engineer: Drive Testing for AI Workflows

Sii Poland; Katowice; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Great Place to Work

Solid financial situation

Contracts with the biggest brands

Centre of internal trainings

Many experts you can learn from

Open and accessible management team

Profit sharing

Passion Sponsorship program

Regular integration events and trips

Comfortable and well-equipped offices

MySii app

Medical care

Job summary

Sii Poland is seeking a Senior QA Engineer to join our team and contribute to building an AI-powered operating system for healthcare marketing. You will own quality across the full stack and help design scalable evaluation pipelines.

In this role you will design evaluation frameworks for LLMs, create automated scoring mechanisms, and monitor data drift and model performance in production. The ideal candidate has 5+ years in software testing, advanced English, and strong Python and

Qualifications

• 5+ years in software testing or quality engineering, with ≥2 years on AI/ML apps.
• Strong Python for backend/AI testing scripts and familiarity with TS/JS for Node.js/Angular testing.
• Experience with evaluation frameworks and selecting metrics for classification or generation quality.
• Familiarity with ML drift monitoring and observability tools.
• Advanced English.

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic harnesses.
• Develop automated scoring mechanisms, assertions, and test suites for agent accuracy and output quality.
• Simulate complex agent workflows to uncover edge cases in autonomous decision-making.
• Collaborate to set up monitoring for data drift, concept drift, and model performance in production.
• Implement telemetry and alerting to track latency, token usage, and cost efficiency.

Skills

Python testing scripts

JavaScript/TypeScript testing

English proficiency

LLM evaluation frameworks

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Datadog

Grafana

Job description

Sii Poland is seeking a Senior QA Engineer to join our team and contribute to building an AI-powered operating system for healthcare marketing. You will own quality across the full stack and help design scalable evaluation pipelines.

In this role you will design evaluation frameworks for LLMs, create automated scoring mechanisms, and monitor data drift and model performance in production. The ideal candidate has 5+ years in software testing, advanced English, and strong Python and