AI/ML QA Engineer — GenAI & LLM Testing

Apptad Inc; Frisco (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

Apptad Inc. in Bellevue, WA and Frisco, TX is hiring a QA Engineer for AI products. You will design and run QA strategies for AI/ML features, validate non-deterministic outputs, and ensure data pipelines quality.

You will build automated test suites, implement evaluation frameworks, test prompts, APIs, and integrate tests into CI/CD. Strong Python, SQL, and experience with RAG/LLM apps required.

Qualifications

• Experienced in testing AI/ML features and AI product workflows.
• Proficient in Python and data validation with SQL.
• Familiar with AI evaluation metrics and non-deterministic outputs.

Responsibilities

• Design and execute QA plans for AI/ML features, prompts, APIs, and app behavior.
• Build and maintain automated test suites for functional, regression, and API testing.
• Test LLM-based products like chatbots, copilots, and RAG apps.
• Evaluate model outputs for accuracy, bias, hallucinations, and edge cases.
• Develop evaluation frameworks, datasets, and scoring methods.
• Conduct prompt regression tests and validate upgrades against baselines.
• Perform adversarial testing for safety, security, and reliability.
• Validate data pipelines, quality, schemas, and drift detection.
• Collaborate with Data Scientists to set quality metrics and criteria.
• Test AI services for latency, scalability, and performance under load.
• Integrate automated tests into CI/CD pipelines.
• Develop Python test harnesses and data validation utilities.
• Use SQL for data validation and backend verification.

Skills

LLM testing

Python

Test automation

SQL

AI evaluation

RAG apps

Chatbots

Copilots

AI agents

Tools

Ragas

DeepEval

LangSmith

Promptfoo

OpenAI Evals

TruLens

Job description

Apptad Inc. in Bellevue, WA and Frisco, TX is hiring a QA Engineer for AI products. You will design and run QA strategies for AI/ML features, validate non-deterministic outputs, and ensure data pipelines quality.

You will build automated test suites, implement evaluation frameworks, test prompts, APIs, and integrate tests into CI/CD. Strong Python, SQL, and experience with RAG/LLM apps required.