AI/LLM QA Engineer - Data Quality & Validation

Sustainment Technologies Inc.; Austin (TX); On-site; USD 90,000 - 130,000; Full time

Benefits offered by this job

Medical, dental, vision coverage

Paid time off

401K matching

Job summary

Sustainment Technologies Inc. is seeking a QA Engineer to ensure reliability and accuracy of AI Agents, focusing on data quality, model evaluation, and regression testing in an AWS environment.

You will design automated QA frameworks, curate ground-truth datasets, and build dashboards to track LLM quality over time, collaborating with ML engineers and product teams to meet strict quality benchmarks.

Qualifications

• 3+ years in software testing and quality assurance.
• 2+ years with a focus on ML evaluation, NLP, LLMs, VLMs, etc.
• Deep understanding of LLM data quality challenges and common failure modes.
• Experience designing automated tests for AI/ML models.
• Familiarity with Python and testing frameworks such as PyTest, Hypothesis, or similar.
• Knowledge of evaluation metrics for LLMs (DeepEval, MLflow, LangSmith, or similar).
• Hands-on experience with automated data validation techniques.
• Strong debugging and analytical skills.
• Experience creating or working with labeled evaluation datasets (“golden” sets) for model evaluation.
• Working knowledge of evaluation metrics for structured information extraction: field-level precision, recall, and F1; exact vs. fuzzy matching; numeric tolerance; and alignment of repeated or nested records.
• Experience translating ambiguous business requirements into precise, documented field definitions in collaboration with non-technical subject-matter experts.

Responsibilities

• Design and run regression test suites for LLM evaluation.
• Identify and track LLM failure modes, including hallucinations, biases, factual inconsistencies, and logical errors.
• Design data-quality checks to assess training and test datasets.
• Automate LLM performance monitoring using advanced metrics and validation strategies.
• Apply best practices for prompt-engineering testing, fine-tuning validation, and output-consistency analysis.
• Collaborate with ML engineers, data scientists, and product teams to align on quality benchmarks.
• Work within an AWS ecosystem, leveraging services such as EKS, S3, SageMaker, or Databricks for model testing and evaluation.
• Build tools and dashboards to track LLM quality over time.
• Curate and version the ground-truth datasets that serve as the accuracy baseline for document parsing, and translate business and domain requirements into written, testable field definitions (partnering with the labeling team on annotation guidelines).
• Evaluate structured extraction from real business documents (multi-page PDFs, scans, spreadsheets) by scoring model output field-by-field against ground truth, with tolerance-aware comparison for numbers, dates, free text, and repeated structures.
• Maintain the ground-truth corpus as a versioned, evolving test asset: keep existing annotations valid as extraction schemas change, preserve dataset provenance, and grow the corpus from real production failures so every customer-reported miss becomes a permanent regression case.
• Calibrate and validate automated scoring itself; confirm that semantic/LLM-judge scoring agrees with human judgment.

Skills

Software testing

ML evaluation

NLP/LLMs

Automated tests

Python testing frameworks

Evaluation metrics for LLMs

Ground-truth datasets

Debugging

Data quality challenges

Field definitions

Tools

Datadog

Kubernetes

Tilt

PostgreSQL

SageMaker

EKS

Databricks

Job description

Sustainment Technologies Inc. is seeking a QA Engineer to ensure reliability and accuracy of AI Agents, focusing on data quality, model evaluation, and regression testing in an AWS environment.

You will design automated QA frameworks, curate ground-truth datasets, and build dashboards to track LLM quality over time, collaborating with ML engineers and product teams to meet strict quality benchmarks.