Junior AI QA Engineer

Simple Staffing; Irvine (CA); On-site; USD 33,000 - 41,000; Full time

Job summary

Tax Relief Advocates is hiring a Junior AI QA Engineer to test AI-driven apps, workflows, and data extraction. You’ll join our team to validate prompts, AI outputs, and business-rule adherence while building regression test suites and documenting defects for developers.

You’ll work on AI-driven web apps, APIs, dashboards, and backend processes, collaborating with AI/ML engineers and product teams in a fast-paced, values-driven environment.

Qualifications

• 0–2 years of software QA or related technical experience.
• Basic understanding of software testing concepts and SDLC.
• Familiarity with REST APIs and JSON.
• Experience or coursework using API testing tools like Postman.
• Strong attention to detail and ability to reproduce problems.
• Interest in AI/LLM applications and willingness to learn evaluation techniques.
• Ability to document test scenarios and defects clearly.

Responsibilities

• Execute manual and automated test cases for AI-enabled applications and workflows.
• Build and maintain regression test suites under guidance.
• Test AI responses for accuracy, relevance, consistency, and adherence to business rules.
• Create test datasets covering normal, edge, malformed, missing, and adversarial inputs.
• Compare expected vs. actual AI outputs and document discrepancies.
• Identify hallucinations, unsupported conclusions, and extraction errors.
• Test prompts and changes to catch regressions before release.
• Perform functional testing of web apps, APIs, dashboards, and backend workflows.
• Validate API requests/responses using tools like Postman.
• Reproduce defects with clear steps and logs for developers.
• Validate fixes in development/staging before production.
• Maintain QA documentation, test cases, results, and release validation records.
• Collaborate with AI QA Engineer, AI/ML engineers, and product teams.

Skills

QA basics

REST APIs

JSON

Postman

Attention to detail

AI evaluation

Tools

Postman

OpenAI API

Playwright

Selenium

LangSmith

Langfuse

Job description

At Tax Relief Advocates, we’re committed to serving our clients and community with integrity and empathy. We balance hard work with a vibrant, energetic culture, aiming to make a meaningful impact nationwide. Our mission is to guide people through the tax resolution process with trust, education, and care.

About the Role

Tax Relief Advocates is looking for a Junior AI QA Engineer to help us test and validate the AI-driven applications, workflows, and features we build. You'll work alongside our AI QA Engineer and development team, combining traditional software QA with AI-specific testing: prompt and output validation, regression testing, hallucination detection, edge-case testing, and evaluation of AI responses. In this role, you'll help test things like AI-driven workflows, document and data extraction, chat and agent behavior, business rules and decision logic, integrations, and the React/API applications that surround our AI systems.

What You'll Do

• Execute manual and automated test cases for AI-enabled applications and workflows.
• Work under the direction of the AI QA Engineer to build and maintain regression test suites.
• Test AI responses for accuracy, relevance, consistency, completeness, and adherence to business rules.
• Create test datasets covering normal cases, edge cases, malformed inputs, missing information, and adversarial scenarios.
• Compare expected vs. actual AI outputs and document discrepancies.
• Identify hallucinations, unsupported conclusions, incorrect classifications, extraction errors, and inconsistent AI behavior.
• Test prompts and prompt changes to catch regressions before they ship.
• Perform functional testing of web applications, APIs, dashboards, and backend workflows.
• Validate API requests and responses using tools such as Postman.
• Perform regression testing before releases.
• Reproduce reported defects and provide developers with clear repro steps, logs, screenshots, inputs, and expected behavior.
• Validate fixes in development/staging environments before production releases.
• Help maintain QA documentation, test cases, results, defect tracking, and release validation records.
• Collaborate closely with our AI QA Engineer, AI/ML engineers, full-stack developers, and product teams.

What We're Looking For

We're not expecting you to already be an ML engineer. We're looking for someone who understands the fundamental difference between deterministic software and probabilistic AI, and who's excited to dig into concepts like:

• LLM prompts and responses
• Hallucinations, and grounded vs. unsupported responses
• Structured output / JSON validation
• RAG (retrieval-augmented generation) at a basic level
• Context windows
• Prompt injection and basic AI security testing
• Regression testing for AI, and accuracy/consistency measurement
• Testing AI against business rules

Required Qualifications

• 0–2 years of software QA, testing, development, data analysis, or related technical experience.
• Basic understanding of software testing concepts and the software development lifecycle.
• Familiarity with REST APIs and JSON.
• Experience or coursework using Postman or similar API testing tools.
• Strong attention to detail and ability to systematically reproduce problems.
• Interest in AI/LLM applications and willingness to learn AI evaluation techniques.
• Ability to document test scenarios and defects clearly.

Nice to Have

• Experience testing LLM or generative AI applications.
• Playwright, Cypress, Selenium, or similar automation experience.
• SQL knowledge.
• Familiarity with OpenAI/Claude/Gemini APIs.
• Experience with LangSmith, Langfuse, or other LLM observability/evaluation platforms.
• Experience testing document extraction, OCR, classification, RAG, or conversational AI.

Why Join Us?

At TRA, you’ll be part of a high-impact, forward-thinking team revolutionizing tax resolution through innovative AI and automation. You'll take on exciting challenges, collaborate with top talent, and play a central role in delivering client-first solutions at scale.

We’re not just building smarter systems—we’re building a better experience for every person who turns to us for help.

Location: Irvine, CA - onsite

Compensation: $27/Hour

Position Type: Full-time, permanent