Senior AI QA Engineer: LLM Testing & AI Automation

Sora Union Group; Polska; Hybrid; PLN 180,000 - 300,000; Full time

Job summary

Sora Union is seeking an experienced Senior AI QA Engineer to join our globally distributed team. You’ll test AI-enabled products, work with developers, product managers, designers and customer service to ensure releases deliver the highest quality for users.

You’ll design AI/LLM tests, build testing agents, validate MCP tools, automate workflows with Playwright MCP, and perform API, data, and exploratory testing. A CS or math degree and 7+ years in QA are expected.

Qualifications

• Degree in computer science or mathematics with 7+ years QA experience.
• Hands-on AI/LLM testing experience and evaluating AI-generated responses.
• Experience with Claude and ChatGPT for testing and automation.
• Experience with Playwright and MCP-based automation to validate workflows.

Responsibilities

• AI and LLM testing to assess accuracy, relevance, and failure cases.
• Build and maintain AI-powered testing agents and test harnesses.
• Test MCP tools, validate input/output, and handle error scenarios.
• Automate testing using Claude, ChatGPT, Playwright MCP.
• Develop test plans, cases, and regression suites.
• Perform API and data validation, and report defects with evidence.
• Collaborate with engineering and product teams to improve releases.

Skills

AI/LLM testing

QA engineering

Cross-functional collaboration

Technical communication

Education

Bachelors in computer science or mathematics

Tools

Playwright MCP

Playwright

Postman

Jira

SQL

GitHub

Job description

Sora Union is seeking an experienced Senior AI QA Engineer to join our globally distributed team. You’ll test AI-enabled products, work with developers, product managers, designers and customer service to ensure releases deliver the highest quality for users.

You’ll design AI/LLM tests, build testing agents, validate MCP tools, automate workflows with Playwright MCP, and perform API, data, and exploratory testing. A CS or math degree and 7+ years in QA are expected.