Senior AI QA Engineer — LLM & API Testing Leader

Sora Union Group; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

Sora Union Group is a professional services company specializing in brand design and engineering services, with a globally distributed team. We seek an experienced QA engineer to design and execute tests for AI/LLM outputs, build testing agents, and validate MCP tools and integrations.

You will develop test plans, regression suites, and automated tests for web apps, APIs, and end-to-end workflows to ensure data integrity and release confidence.

Qualifications

• Degree in computer science or mathematics.
• 7+ years of professional QA experience.
• Hands-on AI/LLM testing experience.
• Experience with Claude, ChatGPT, or similar AI tools for testing.
• Experience with Playwright and MCP-based automation.
• Working knowledge of REST APIs and SQL.
• Familiarity with Jira, TestRail, Azure DevOps, or GitHub.
• Understanding of Agile/Scrum methodologies.
• English upper-intermediate or higher.

Responsibilities

• Design and execute tests evaluating LLM-generated responses for accuracy and consistency.
• Build and maintain AI-powered testing agents and test harnesses.
• Test MCP tools and integrations, including error scenarios.
• Develop test plans, regression suites, and manual testing checklists.
• Design and maintain automated tests for web apps, APIs, and end-to-end workflows.
• Test backend APIs, verify data integrity, and validate system integrations.
• Reproduce, document, and triage defects with actionable evidence.
• Partner with engineering and product teams to assess risks and improve release confidence.

Skills

AI/LLM testing

QA testing

Automation

Playwright

REST APIs

SQL

Agile/Scrum

English communication

Education

Bachelor's degree in Computer Science or Mathematics

Tools

Playwright

MCP automation

Jira

TestRail

Azure DevOps

GitHub

Job description

Sora Union Group is a professional services company specializing in brand design and engineering services, with a globally distributed team. We seek an experienced QA engineer to design and execute tests for AI/LLM outputs, build testing agents, and validate MCP tools and integrations.

You will develop test plans, regression suites, and automated tests for web apps, APIs, and end-to-end workflows to ensure data integrity and release confidence.