Senior AI QA Engineer - Automation & Evaluation

Sii Poland; Rzeszów; On-site; PLN 180,000 - 260,000; Full time

Benefits offered by this job

Great team

Competitive salary

Training programs

Medical care

Office in Rzeszów

Job summary

Sii Poland is seeking a Senior QA Engineer to build and own automated testing and AI-evaluation pipelines for an AI-driven healthcare marketing platform. You’ll design, implement, and monitor test frameworks, ensuring end-to-end quality across frontend, backend, and AI microservices.

You will collaborate with cross-functional teams to scale AI evaluation, detect drift, and optimize latency and cost while delivering robust test automation in Python and JavaScript/TypeScript.

Qualifications

• 5+ years in software testing with AI/ML emphasis.
• Proficient in Python for test scripts and TypeScript for Node.js/Angular testing.
• Experience with AI evaluation frameworks and telemetry.
• Strong English communication and collaboration skills.

Responsibilities

• Design, build, and maintain evaluation frameworks for LLMs and agentic harnesses.
• Develop automated scoring, assertions, and test suites for accuracy, reasoning, and tool use.

Skills

Python

JavaScript/TypeScript

QA Automation

AI/ML Testing

Test Strategy

Tools

LangSmith

Langfuse

EvidentlyAI

Ragas

MLflow

Datadog

Grafana

Job description

Sii Poland is seeking a Senior QA Engineer to build and own automated testing and AI-evaluation pipelines for an AI-driven healthcare marketing platform. You’ll design, implement, and monitor test frameworks, ensuring end-to-end quality across frontend, backend, and AI microservices.

You will collaborate with cross-functional teams to scale AI evaluation, detect drift, and optimize latency and cost while delivering robust test automation in Python and JavaScript/TypeScript.