Senior AI QA Engineer: AI Evaluation & Reliability

Sii Poland; Szczecin; On-site; PLN 180,000 - 300,000; Full time

Benefits offered by this job

Great Place to Work

Solid financial situation

Centre of internal trainings

Many experts to learn from

Open management team

Profit sharing

Sii sponsorship programs

Job summary

Sii Poland is seeking a Senior QA Engineer to join our team and contribute to an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems, ensuring the stability of a full-stack architecture and scalable AI evaluation pipelines.

You will design and scale evaluation frameworks for LLMs, develop automated scoring, and implement telemetry to monitor drift, latency, and output quality in production.

Qualifications

• Minimum 5 years in software testing or quality engineering with at least 2 years in AI/ML applications, LLM workflows, or agentic systems
• Proficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems
• Experience with evaluation frameworks and metrics selection for classification or generation-quality problems
• Familiarity with ML drift monitoring concepts, metrics tracking, and observability tools
• Strong system-level design skills to build production-grade test infrastructure
• Advanced level of English

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic harnesses
• Develop automated scoring mechanisms, assertions, and test suites for agent accuracy and hallucination rates
• Simulate multi-step agent workflows to uncover edge cases in autonomous decision-making
• Set up monitoring for data drift, concept drift, and model performance degradation
• Implement telemetry and alerting for latency, token usage, and cost efficiency
• Own end-to-end testing strategy across Angular frontend, Node.js backend, and Python AI microservices
• Write scalable automated test scripts in Python and JavaScript/TypeScript
• Perform API, integration, and performance/load testing for microservices
• Integrate AI evaluation into CI/CD pipelines and enforce quality gates
• Define and track KPIs for software reliability and AI model behavior

Skills

Python

JavaScript/TypeScript

QA Testing

AI/ML Testing

English Proficiency

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Job description

Sii Poland is seeking a Senior QA Engineer to join our team and contribute to an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems, ensuring the stability of a full-stack architecture and scalable AI evaluation pipelines.

You will design and scale evaluation frameworks for LLMs, develop automated scoring, and implement telemetry to monitor drift, latency, and output quality in production.