QA Engineer, LLM & GenAI Testing

Apptad Inc; Frisco (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Apptad Inc. is seeking an AI test engineer to design and execute test plans for AI/ML features, including model outputs, prompts, and integrated app behavior.

You will build automated test suites for functional, regression, integration, and API testing, and collaborate with data scientists to define acceptance criteria and quality metrics. You will evaluate model outputs for accuracy, bias, and edge cases, develop evaluation datasets, and perform adversarial testing to surface safety and

Qualifications

• Experience testing LLM/GenAI products and evaluating non-deterministic outputs.
• Strong Python skills for writing eval scripts and data validation.
• Proficiency with SQL and data quality validation techniques.
• Familiarity with AI evaluation frameworks and model safety testing.
• Experience in building automated test suites and CI/CD integration.

Responsibilities

• Design and execute test plans for AI/ML features and model outputs.
• Build automated test suites for functional, regression, API testing.
• Evaluate model outputs for accuracy, bias, and edge cases.
• Develop evaluation datasets and scoring rubrics for benchmarks.
• Perform adversarial testing to surface safety and robustness issues.
• Validate data pipelines feeding AI models and collaborate with ML teams.
• Test latency, scalability, and reliability of AI services under load.
• Contribute to CI/CD pipelines with model-evaluation tests.

Skills

LLM/GenAI testing

Python

AI evaluation frameworks

SQL

Test automation

Tools

LangSmith

Promptfoo

OpenAI Evals

TruLens

Job description

Apptad Inc. is seeking an AI test engineer to design and execute test plans for AI/ML features, including model outputs, prompts, and integrated app behavior.

You will build automated test suites for functional, regression, integration, and API testing, and collaborate with data scientists to define acceptance criteria and quality metrics. You will evaluate model outputs for accuracy, bias, and edge cases, develop evaluation datasets, and perform adversarial testing to surface safety and