Senior Automation Test Engineer – AI Systems (f/m/x)

Sii Poland; Kraków; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Great Place to Work

Profit sharing

Centre of internal trainings

Open and accessible management team

Medical care

Job summary

Sii Poland is seeking a Senior QA Engineer to lead testing and evaluation for an AI-powered healthcare marketing platform. You will own end-to-end QA across frontend Angular, Node.js backend, and Python AI microservices, building scalable evaluation pipelines and drift monitoring systems.

You will design automated tests, frameworks for LLMs, and monitoring for data/model drift while collaborating with cross-functional teams to ensure robust, production-grade quality and measurable KPIs.

Qualifications

• 5+ years in software testing or quality engineering, with ≥2 years in AI/ML testing or LLM workflows.
• Proficiency in Python for backend/AI testing and JavaScript/TypeScript for Node.js/Angular testing.
• Experience with LangSmith, Langfuse, EvidentlyAI, Ragas or MLflow and selecting metrics for classification or generation-quality problems.
• Familiarity with ML drift monitoring concepts, metrics tracking, and observability tools like Datadog or Grafana.
• Strong system-level design skills to build production-grade test and evaluation infrastructure from the ground up.
• Advanced level of English.

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic harnesses.
• Develop automated scoring, assertions, and test suites to measure accuracy and hallucinations.
• Simulate multi-step agent workflows to uncover edge cases in autonomous decision-making.
• Set up monitoring frameworks for data drift and model performance in production.
• Implement telemetry and alerting for latency, token usage, cost, and quality.
• Own end-to-end testing strategy across Angular frontend, Node.js backend, and Python AI microservices.
• Write clean, maintainable automated test scripts in Python and JavaScript/TypeScript.
• Perform API, integration, and performance/load testing for high-throughput microservices.
• Integrate AI evaluation into CI/CD pipelines for quality gates.
• Define and track KPIs for software reliability and AI model behavior.

Skills

Python

JavaScript/TypeScript

English proficiency

AI/ML testing

LLM evaluation

ML drift monitoring

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Job description

We are looking for a Senior QA Engineer to join our team and contribute to building an AI-powered operating system for healthcare marketing. In this role, you will go beyond traditional end-to-end testing, owning quality, evaluation, and monitoring for complex AI systems. You will work on real-world production-grade products that create monetizable value, ensuring the stability of our full-stack architecture while designing and scaling AI evaluation pipelines and ML drift monitoring systems. If you are passionate about developing robust test and evaluation frameworks for modern systems powered by AI and ML, we want to hear from you.

Your tasks

• Designing, building, and maintaining evaluation frameworks for Large Language Models (LLMs) and custom agentic harnesses
• Developing automated scoring mechanisms, assertions, and test suites to measure agent accuracy, reasoning reliability, hallucination rates, and tool-use correctness
• Simulating complex, multi-step agent workflows to uncover edge cases in autonomous decision-making
• Collaborating with engineering teams to set up robust monitoring frameworks for data drift, concept drift, and model performance degradation in production
• Implementing telemetry and alerting systems to track latency, token usage, cost efficiency, and output quality over time
• Owning the end-to-end testing strategy across our entire stack, ensuring seamless integration between the Angular frontend, Node.js backend services, and Python-based AI microservices
• Writing clean, maintainable, and scalable automated test scripts in Python and JavaScript/TypeScript
• Performing rigorous API testing, integration testing, and performance/load testing for high-throughput microservices
• Integrating AI evaluation and testing harnesses into our CI/CD pipelines to ensure continuous quality gate enforcement before deployment
• Defining and tracking key quality metrics (KPIs) for software reliability and AI model behavior

Requirements

• Minimum 5 years of experience in software testing or quality engineering, with at least 2 years focused on testing or evaluating AI/ML applications, LLM workflows, or agentic systems
• Proficiency in Python for backend/AI testing scripts and familiarity with JavaScript/TypeScript for Node.js and Angular testing ecosystems
• Experience with evaluation frameworks such as LangSmith, Langfuse, EvidentlyAI, Ragas, or MLflow, and the ability to select appropriate metrics for classification or generation-quality problems
• Familiarity with ML drift monitoring concepts, metrics tracking, and observability tools like Datadog or Grafana
• Strong system-level design skills and the ability to build production-grade test and evaluation infrastructure from the ground up
• Advanced level of English

Nice to have

• Understanding of agentic design patterns and their decomposition into testable units
• Experience working under the regulatory conditions of healthcare systems

Job no. JOB-GTXQF

Sii ensures that all hiring decisions are made solely on the basis of qualifications and competence. We are committed to equal and fair treatment of all, regardless of legally protected characteristics. At Sii, we promote a diverse and inclusive work environment, in full compliance with applicable anti-discrimination laws.

Benefits For You

• Great Place to Work
• Solid financial situation
• Contracts with the biggest brands
• Centre of internal trainings
• Many experts you can learn from
• Open and accessible management team
• Profit sharing
• Passion Sponsorship program
• Regular integration events and trips
• Comfortable and well-equipped offices
• MySii app
• Medical care