QA & Automation Engineer with AI tools experience

Robotics Prcocess Automation, LLC; Austin (TX); On-site; USD 55,000 - 117,000; Full time

Job summary

Robotics Process Automation, LLC is looking for a QA & Automation Engineer with experience in AI tools to develop test suites for multi-agent workflows and APIs in Austin, TX. The role involves automating regression tests for LLM-driven features. Candidates must have a strong background in automation for AI/LLM systems and proficiency in API/E2E testing using tools like Playwright and Cypress. This position offers an hourly rate depending on experience and may include extensions beyond the initial 12 months.

Qualifications

• Experience with test automation in AI/LLM systems, including flakiness and hallucinations.
• Proficiency in API/E2E testing using Playwright, Cypress, and Pytest.
• Familiarity with agent reliability evaluation frameworks.

Responsibilities

• Develop AI-aligned test suites for multi-agent workflows and APIs.
• Work with Test Agent to auto-generate and optimize test coverage.
• Automate regression tests for LLM-driven features.

Skills

AI-aligned test suites

LLM system

ML model testing

API/E2E testing (Playwright, Cypress, Pytest)

Tools

Playwright

Cypress

Pytest

Job description

QA & Automation Engineer with AI tools experience

Location: Austin, TX

Duration: 12Months+ Extension

Hourly Rate: Depending on Experience (DOE)

Required Skills

• AI-aligned test suites
• LLM system
• ML model testing
• API/E2E testing (Playwright, Cypress, Pytest)

Responsibilities

• Develop AI-aligned test suites for multi-agent workflows, cross-cloud integrations, and APIs.
• Work with Test Agent to auto-generate and optimize test coverage.
• Automate regression and reliability tests for LLM-driven features.

Requirements

• Experience with test automation in AI/LLM systems (flakiness, non-determinism, hallucinations).
• Proficiency in API/E2E testing (Playwright, Cypress, Pytest).
• Familiarity with agent reliability evaluation frameworks.

Preferred

• Experience with automated defect classification and ML model testing.

ROBOTIC PROCESS AUTOMATION LLC is an equal opportunity employer inclusive of female, minority, disability and veterans, (M/F/D/V). Hiring, promotion, transfer, compensation, benefits, discipline, termination and all other employment decisions are made without regard to race, color, religion, sex, sexual orientation, gender identity, age, disability, national origin, citizenship/immigration status, veteran status or any other protected status. ROBOTIC PROCESS AUTOMATION LLC will not make any posting or employment decision that does not comply with applicable laws relating to labor and employment, equal opportunity, employment eligibility requirements or related matters. Nor will ROBOTIC PROCESS AUTOMATION LLC require in a posting or otherwise U.S. citizenship or lawful permanent residency in the U.S. as a condition of employment except as necessary to comply with law, regulation, executive order, or federal, state, or local government contract.