Senior QA Engineer — AI/ML Testing & Evaluation

Sii Poland; Lublin; On-site; PLN 120,000 - 180,000; Full time

Benefits offered by this job

Great Place to Work

Solid financial situation

Contracts with the biggest brands

Centre of internal trainings

Many experts you can learn from

Open and accessible management team

Profit sharing

Passion Sponsorship program

Regular integration events and trips

Comfortable and well-equipped offices

MySii app

Medical care

Job summary

Sii Poland is seeking a Senior QA Engineer to join the team and drive testing and evaluation for an AI-powered operating system for healthcare marketing. You will own quality across AI systems, design evaluation pipelines, and monitor ML drift in production, collaborating with cross-functional teams to ensure robust, scalable test frameworks.

You will implement automated test scripts in Python and JavaScript/TypeScript, perform API and performance testing for high-throughput microservices, and

Qualifications

• Minimum 5 years of experience in software testing or quality engineering.
• At least 2 years focused on AI/ML applications, LLM workflows or agentic systems.
• Proficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems.
• Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas or MLflow.
• Familiarity with ML drift monitoring concepts, metrics tracking, and observability tools like Datadog or Grafana.
• Strong system-level design skills to build production-grade test and evaluation infrastructure.

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and custom agent harnesses.
• Develop automated scoring mechanisms and test suites to measure accuracy, reasoning, hallucination rates, and tool-use correctness.
• Simulate multi-step agent workflows to uncover edge cases in autonomous decision-making.
• Set up robust monitoring for data drift, concept drift, and model performance degradation.
• Implement telemetry and alerting to track latency, token usage, cost, and output quality over time.
• Own end-to-end testing across the stack, ensuring integration between Angular frontend, Node.js backend, and Python AI services.
• Write scalable automated test scripts in Python and JavaScript/TypeScript.
• Perform API, integration, and performance/load testing for high-throughput microservices.
• Integrate AI evaluation into CI/CD pipelines for quality gates before deployment.
• Define and track KPIs for software reliability and AI model behavior.

Skills

Python

JavaScript/TypeScript

AI/ML testing

QA testing

Evaluation frameworks

CI/CD

Observability tools

English proficiency

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Datadog

Grafana

Job description

Sii Poland is seeking a Senior QA Engineer to join the team and drive testing and evaluation for an AI-powered operating system for healthcare marketing. You will own quality across AI systems, design evaluation pipelines, and monitor ML drift in production, collaborating with cross-functional teams to ensure robust, scalable test frameworks.

You will implement automated test scripts in Python and JavaScript/TypeScript, perform API and performance testing for high-throughput microservices, and