AI Quality & Automation Engineer

Patrianna Limited; Polska; On-site; PLN 120,000 - 180,000; Full time

Job summary

Patrianna Limited is seeking an AI QA Engineer to ensure our agents and critical workflows run reliably through automated testing and evaluation. You will design, implement and maintain tests for APIs, web apps, and integrations, and assess model performance against clear criteria.

Collaborating with the AI and QA teams, you will capture defects, build regression cases, and embed quality checks into CI/CD pipelines.

Qualifications

• Strong test automation and programming skills in TypeScript or Python.
• Solid experience with web apps, APIs, Git and CI/CD pipelines.
• Structured approach to evaluating AI output against measurable criteria.
• Ability to reproduce, isolate and pin down bugs quickly with root cause analysis.

Responsibilities

• Build and maintain API, integration and end-to-end regression tests for critical workflows.
• Evaluate AI agents on task completion, response quality, tool use and permissions.
• Integrate quality checks into CI/CD pipelines and create regression cases for bugs.
• Explore workflows with agents, generate tests and reproduce defects for verification.
• Share tools and methods with the AI team and central QA to raise standards.

Skills

Test automation

TypeScript

Python

Web apps

APIs

Git

CI/CD

Tools

Supabase

Vercel

GCP

LLM evaluation frameworks

Job description

Patrianna Limited is seeking an AI QA Engineer to ensure our agents and critical workflows run reliably through automated testing and evaluation. You will design, implement and maintain tests for APIs, web apps, and integrations, and assess model performance against clear criteria.

Collaborating with the AI and QA teams, you will capture defects, build regression cases, and embed quality checks into CI/CD pipelines.