LLM QA Engineer: AI Data Quality & Regression (Contract)

Blackhornvc; Austin (TX); Hybrid; USD 83,000 - 165,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Vision benefits

Paid time off

Company holidays

401K matching

Job summary

Sustainment is seeking a QA Engineer to ensure reliability and robustness of AI Agents in an AWS-based setup. You will design automated QA systems, evaluate ground-truth datasets, and score structured extractions from multi-page documents, adapting to changing schemas and production failures.

Candidate should have 3+ years in software QA with ML/NLP focus, strong Python, PyTest, and data-validation skills, and experience with LLM eval and model monitoring.

Qualifications

• 3+ years in software testing and quality assurance.
• 2+ years with a focus on ML evaluation, NLP, LLMs, VLMs, etc.
• Deep understanding of LLM data quality challenges and common failure modes.

Responsibilities

• Design and run regression test suites for LLM evaluation.
• Identify and track LLM failure modes, including hallucinations, biases, factual inconsistencies, and logical errors.
• Design data-quality checks to assess training and test datasets.
• Automate LLM performance monitoring using advanced metrics and validation strategies.
• Apply best practices for prompt-engineering testing, fine-tuning validation, and output-consistency analysis.
• Collaborate with ML engineers, data scientists, and product teams to align on quality benchmarks.
• Work within an AWS ecosystem, leveraging services such as EKS, S3, SageMaker, or Databricks for model testing and evaluation.
• Build tools and dashboards to track LLM quality over time.
• Curate and version the ground-truth datasets that serve as the accuracy baseline for document parsing, and translate business and domain requirements into written, testable field definitions.
• Evaluate structured extraction from real business documents by scoring model output field-by-field against ground truth, with tolerance-aware comparison for numbers, dates, free text, and repeated structures.
• Maintain the ground-truth corpus as a versioned, evolving test asset: keep existing annotations valid as extraction schemas change, preserve dataset provenance, and grow the corpus from real production failures so every customer-reported miss becomes a permanent regression case.
• Calibrate and validate automated scoring itself; confirm that semantic/LLM-judge scoring agrees with human judgment.

Skills

Python

PyTest

Hypothesis

LangSmith

MLflow

SageMaker

Datadog

Kubernetes

Tilt

OpenAI APIs

Anthropic APIs

Bedrock

.NET

EF Core

MLOps

OCR/Document AI

Tools

Kubernetes

Tilt

Datadog

SageMaker

Databricks

EF Core

Job description

Sustainment is seeking a QA Engineer to ensure reliability and robustness of AI Agents in an AWS-based setup. You will design automated QA systems, evaluate ground-truth datasets, and score structured extractions from multi-page documents, adapting to changing schemas and production failures.

Candidate should have 3+ years in software QA with ML/NLP focus, strong Python, PyTest, and data-validation skills, and experience with LLM eval and model monitoring.