AI Quality Engineer: LLM Testing & Validation

Global Business Ser. 4u; Plano (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Global Business Ser. 4u in Plano, TX is seeking a V&V Engineer for AI-driven testing and validation. You will lead end-to-end QA for enterprise AI apps, including LLM-powered products, RAG pipelines, and agentic workflows.

You will design prompt validation strategies, build Python automation frameworks, and integrate testing into CI/CD to validate model updates and data revisions before release. You will collaborate with client AI engineers, define KPIs, mentor QA staff, and promote

Qualifications

• 10+ years of Quality Engineering and Test Automation in enterprise apps.
• Proficient in validating AI/ML systems, including Generative AI.
• Strong Python and automation framework development.
• Practical experience with prompt validation and agentic workflow testing.
• Knowledge of evaluation metrics: BLEU, ROUGE, embedding similarity, precision, recall, F1-score, and human evaluation.

Responsibilities

• Lead end-to-end quality engineering for enterprise AI applications, including LLM-powered products and agentic workflows.
• Design and execute prompt validation strategies, evaluating LLM responses for accuracy and safety.
• Build automated evaluation pipelines for AI model outputs using BLEU, ROUGE, embeddings, precision, recall, and F1-score.
• Validate agentic systems for correctness, determinism, and failure mode handling.
• Architect Python-based automation frameworks for AI model evaluation, regression testing, and monitoring.
• Integrate AI testing into CI/CD pipelines to validate model updates before release.
• Develop reusable test harnesses for prompt regression, golden-set evaluation, and AI review workflows.
• Perform data validation, bias detection, and fairness analysis across training and inference pipelines.
• Drive robustness testing including adversarial inputs and distribution shift detection.
• Collaborate with client AI engineers and mentor QA staff on AI evaluation methodologies.

Skills

Python

AI/ML validation

Prompt validation

CI/CD integration

Test automation

Data validation

Bias/fairness analysis

TensorFlow

PyTorch

LangChain

LangGraph

LlamaIndex

GitHub Actions

Education

Bachelor's or Master's in CS/DS

Tools

TensorFlow

PyTorch

LangChain

LangGraph

LlamaIndex

MLflow

Weights & Biases

LangSmith

Ragas

DeepEval

TruLens

Jenkins

GitHub Actions

Azure DevOps

Job description

Global Business Ser. 4u in Plano, TX is seeking a V&V Engineer for AI-driven testing and validation. You will lead end-to-end QA for enterprise AI apps, including LLM-powered products, RAG pipelines, and agentic workflows.

You will design prompt validation strategies, build Python automation frameworks, and integrate testing into CI/CD to validate model updates and data revisions before release. You will collaborate with client AI engineers, define KPIs, mentor QA staff, and promote