Senior AI QA Engineer: LLM Testing & Automation Leader

soraunion

United States

Remote

USD 140,000 - 190,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Sora Union in the United States is seeking a Senior AI QA Engineer to ensure high-quality releases across AI-driven products. You will work with engineers, product managers, designers, and customer teams to design tests, validate responses, and report results.

You will lead automation efforts using Claude, ChatGPT, Playwright MCP, and related tools, build testing agents, and develop test plans, covering web apps, APIs, and data flows.

Qualifications

  • A degree in computer science or mathematics with a minimum of 7 years in a professional QA role.
  • Hands-on AI/LLM testing experience, evaluating AI-generated responses and identifying issues.
  • Practical experience using Claude and ChatGPT to support testing and automation.
  • Experience with Playwright MCP or similar automation for workflows.
  • Experience building or maintaining AI-powered testing agents and test harnesses.
  • Strong knowledge of manual, exploratory, functional, regression, and end-to-end testing.
  • Experience with testing web apps and backend APIs, including data flows and integrations.
  • Familiarity with Jira, TestRail, Azure DevOps, or GitHub for defect/test management.
  • Understanding of Agile/Scrum and cross-functional teams.
  • Excellent written and verbal communication with the ability to explain technical findings.

Responsibilities

  • Design and execute tests to evaluate LLM-generated responses for accuracy, relevance, consistency, and potential failure cases.
  • Build, maintain, and improve AI-powered testing agents and test harnesses.
  • Test Model Context Protocol (MCP) tools and integrations, validating behavior and error handling.
  • Use Claude, ChatGPT, Playwright MCP to automate test creation, execution, and reporting.
  • Develop and maintain comprehensive test plans, test cases, regression suites, and checklists.
  • Automate tests for web applications, APIs, and end-to-end workflows.
  • Perform functional and exploratory testing to identify edge cases and data-related issues.
  • Test backend APIs and ensure data integrity with SQL queries.
  • Document defects with evidence and work with development teams for resolution.
  • Collaborate with engineering, product, and other teams to clarify requirements and reduce risk.
  • Identify opportunities to improve QA processes and AI-driven testing approaches.
  • Develop deep product understanding to reflect real-world usage in testing.

Skills

AI/LLM testing
Automation testing
API testing
SQL queries
Cross-functional collaboration
Analytical thinking
Effective communication

Education

BS in computer science or mathematics

Tools

Claude
ChatGPT
Playwright MCP
Selenium
Postman
Jira
TestRail
Azure DevOps
GitHub
GraphQL

Job description

Sora Union in the United States is seeking a Senior AI QA Engineer to ensure high-quality releases across AI-driven products. You will work with engineers, product managers, designers, and customer teams to design tests, validate responses, and report results.

You will lead automation efforts using Claude, ChatGPT, Playwright MCP, and related tools, build testing agents, and develop test plans, covering web apps, APIs, and data flows.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI QA Engineer — LLM & API Testing Leader
Senior AI QA Engineer — LLM & API Testing Leader

Sora Union Group • United States

Remote
USD 120,000 - 170,000
Senior AI QA Engineer for Sora Union
Senior AI QA Engineer for Sora Union

Sora Union • United States

Remote
USD 120,000 - 170,000
Senior AI QA Engineer for Sora Union
Senior AI QA Engineer for Sora Union

soraunion • United States

Remote
USD 140,000 - 190,000
Senior AI Test Automation Engineer Frontline AI Quality
Senior AI Test Automation Engineer Frontline AI Quality

Genuine Parts Company • United States

On-site
USD 120,000 - 155,000
Remote AI QA Engineer for LLMs & Autonomous Agents
Remote AI QA Engineer for LLMs & Autonomous Agents

web • United States

Remote
USD 120,000 - 180,000
Senior QA Engineer & Quality Lead, AI SaaS
Senior QA Engineer & Quality Lead, AI SaaS

Devconnectplatform • United States

Remote
USD 120,000 - 150,000
LLM QA Engineer | AI Testing & Automation
LLM QA Engineer | AI Testing & Automation

Valiant Technologies LLC • Chicago (IL)

On-site
USD 70,000 - 110,000
Senior AI Quality Engineer - SaaS Quality Leader
Senior AI Quality Engineer - SaaS Quality Leader

affinipay1 • United States

Hybrid
USD 53,000 - 81,000
Flexible Time Off
13 paid public holidays
Paid sick leave
+2
AI Test Engineer (LLM and AI QA)
AI Test Engineer (LLM and AI QA)

Synoptek • United States

Remote
USD 90,000 - 120,000
Senior QA Engineer: AI-Driven Testing & Quality Champion
Senior QA Engineer: AI-Driven Testing & Quality Champion

Ciandt • United States

Remote
USD 70,000 - 110,000
Health and dental insurance
Meal/food allowance
Continuous learning (CI&T University)
+1