Senior AI QA Engineer: ML Testing & Evaluation

Sii Poland; Bydgoszcz; On-site; PLN 180,000 - 320,000; Full time

Benefits offered by this job

Great Place to Work

Centre of internal trainings

Profit sharing

Medical care

Open and accessible management team

Job summary

Sii Poland is seeking a Senior QA Engineer to join our team in building an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems across the stack, from Angular frontend to Python microservices.

You will design evaluation pipelines, implement automated tests in Python and JavaScript/TypeScript, and work with cross-functional teams to ensure scalable, reliable systems in production. Fluency in English is required.

Qualifications

• Minimum 5 years in software testing with AI/ML focus.
• Proficiency in Python for backend/AI testing tasks.
• Experience with evaluation frameworks (LangSmith, Langfuse, EvidentlyAI, MLflow).
• Familiarity with ML drift monitoring and observability tools (Datadog, Grafana).
• Strong system-level design skills and English fluency.

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic systems.
• Develop automated scoring mechanisms, assertions, and test suites.
• Simulate complex agent workflows to reveal edge cases in autonomous decisions.
• Collaborate to implement monitoring for data drift and model performance in production.
• Implement telemetry and alerting for latency, token usage, and cost efficiency.
• Own end-to-end testing across Angular frontend, Node.js backend, and Python microservices.
• Write clean automated tests in Python and JavaScript/TypeScript.
• Perform API, integration, and performance/load testing for high-throughput services.
• Integrate AI evaluation into CI/CD pipelines for quality gates.
• Define and track KPIs for software reliability and AI behavior.

Skills

Python testing scripts

JavaScript/TypeScript testing

AI/ML evaluation

English communication

Job description

Sii Poland is seeking a Senior QA Engineer to join our team in building an AI-powered operating system for healthcare marketing. You will own quality, evaluation, and monitoring for complex AI systems across the stack, from Angular frontend to Python microservices.

You will design evaluation pipelines, implement automated tests in Python and JavaScript/TypeScript, and work with cross-functional teams to ensure scalable, reliable systems in production. Fluency in English is required.