LLM QA Engineer – AI Data Quality & Testing

Sustainment; Austin (TX); On-site; USD 110,000 - 170,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

Paid Time Off

Company Holidays

401K Matching

Job summary

Sustainment is seeking a QA Engineer to ensure reliability and robustness of AI Agents in an AWS environment. You will design automated QA for LLM evaluation, data validation, and ground-truth curation for document parsing across multi-page PDFs and scans.

You will collaborate with ML engineers and product teams, build dashboards, and tune scoring to align automated results with human judgments. This is a contract opportunity in the US.

Qualifications

• 3+ years in software testing and quality assurance.
• 2+ years with ML evaluation, NLP, LLMs, VLMs, etc.
• Deep understanding of LLM data quality challenges and common failure modes.
• Experience designing automated tests for AI/ML models.
• Familiarity with Python and testing frameworks such as PyTest, Hypothesis, or similar.
• Knowledge of evaluation metrics for LLMs (DeepEval, MLflow, LangSmith, or similar).
• Hands-on experience with automated data validation techniques.
• Strong debugging and analytical skills.
• Experience creating or working with labeled evaluation datasets ("golden" sets).
• Working knowledge of evaluation metrics for structured information extraction.

Responsibilities

• Design and run regression test suites for LLM evaluation.
• Identify and track LLM failure modes, including hallucinations, biases, factual inconsistencies, and logical errors.
• Design data-quality checks to assess training and test datasets.
• Automate LLM performance monitoring using advanced metrics and validation strategies.
• Apply best practices for prompt-engineering testing, fine-tuning validation, and output-consistency analysis.
• Collaborate with ML engineers, data scientists, and product teams to align on quality benchmarks.
• Work within an AWS ecosystem, leveraging services such as EKS, S3, SageMaker, or Databricks for model testing and evaluation.
• Build tools and dashboards to track LLM quality over time.
• Curate and version the ground-truth datasets that serve as the accuracy baseline for document parsing.
• Evaluate structured extraction from real business documents by scoring model output field-by-field against ground truth.
• Maintain the ground-truth corpus as a versioned, evolving test asset and grow it from real production failures.
• Calibrate and validate automated scoring itself; ensure semantic/LLM-judge scoring matches human judgment.

Skills

QA testing

ML evaluation

NLP/LLM

Python

PyTest

Hypothesis

Evaluation metrics

Data validation

Debugging

Ground-truth datasets

Tools

Datadog

SageMaker

EKS

Postgres

Docker

Kubernetes

Job description

Sustainment is seeking a QA Engineer to ensure reliability and robustness of AI Agents in an AWS environment. You will design automated QA for LLM evaluation, data validation, and ground-truth curation for document parsing across multi-page PDFs and scans.

You will collaborate with ML engineers and product teams, build dashboards, and tune scoring to align automated results with human judgments. This is a contract opportunity in the US.