V&V Engineer – AI-Driven Testing & Validation

Global Business Ser. 4u; Plano (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

Global Business Ser. 4u in Plano, TX is seeking a V&V Engineer for AI-driven testing and validation. You will lead end-to-end QA for enterprise AI apps, including LLM-powered products, RAG pipelines, and agentic workflows.

You will design prompt validation strategies, build Python automation frameworks, and integrate testing into CI/CD to validate model updates and data revisions before release. You will collaborate with client AI engineers, define KPIs, mentor QA staff, and promote

Qualifications

• 10+ years of Quality Engineering and Test Automation in enterprise apps.
• Proficient in validating AI/ML systems, including Generative AI.
• Strong Python and automation framework development.
• Practical experience with prompt validation and agentic workflow testing.
• Knowledge of evaluation metrics: BLEU, ROUGE, embedding similarity, precision, recall, F1-score, and human evaluation.

Responsibilities

• Lead end-to-end quality engineering for enterprise AI applications, including LLM-powered products and agentic workflows.
• Design and execute prompt validation strategies, evaluating LLM responses for accuracy and safety.
• Build automated evaluation pipelines for AI model outputs using BLEU, ROUGE, embeddings, precision, recall, and F1-score.
• Validate agentic systems for correctness, determinism, and failure mode handling.
• Architect Python-based automation frameworks for AI model evaluation, regression testing, and monitoring.
• Integrate AI testing into CI/CD pipelines to validate model updates before release.
• Develop reusable test harnesses for prompt regression, golden-set evaluation, and AI review workflows.
• Perform data validation, bias detection, and fairness analysis across training and inference pipelines.
• Drive robustness testing including adversarial inputs and distribution shift detection.
• Collaborate with client AI engineers and mentor QA staff on AI evaluation methodologies.

Skills

Python

AI/ML validation

Prompt validation

CI/CD integration

Test automation

Data validation

Bias/fairness analysis

TensorFlow

PyTorch

LangChain

LangGraph

LlamaIndex

GitHub Actions

Education

Bachelor's or Master's in CS/DS

Tools

TensorFlow

PyTorch

LangChain

LangGraph

LlamaIndex

MLflow

Weights & Biases

LangSmith

Ragas

DeepEval

TruLens

Jenkins

GitHub Actions

Azure DevOps

Job description

Role: V&V Engineer – AI-Driven Testing & Validation

Work location: Plano, TX

Job Description

AI/ML & LLM Development/Validation

Lead end-to-end quality engineering for enterprise AI applications, including LLM-powered products, RAG pipelines, and agentic workflows.

Design and execute prompt validation strategies, evaluating LLM responses for accuracy, semantic relevance, hallucination risk, and safety compliance.

Build automated evaluation pipelines for AI model outputs using metrics such as BLEU, ROUGE, embedding-based similarity, precision, recall, and F1-score.

Validate agentic systems (tool use, multi-step reasoning, planner-executor workflows) for correctness, determinism, and failure mode handling.

Test Automation & Frameworks

Architect and maintain Python-based automation frameworks for AI/ML model evaluation, regression testing, and continuous model quality monitoring.

Integrate AI testing into CI/CD pipelines, enabling automated evaluation of model updates, prompt changes, and dataset revisions before release.

Develop reusable test harnesses for prompt regression, golden-set evaluation, A/B comparison of model versions, and human-in-the-loop review workflows.

Data Quality, Bias & Fairness

Perform AI data validation across training and inference pipelines using exploratory data analysis (EDA), schema validation, and cross-validation techniques.

Conduct bias detection and fairness analysis across demographic and contextual slices to ensure responsible AI outcomes.

Drive model robustness testing, including adversarial inputs, distribution shift detection, and stress testing under edge cases.

Establish regression testing standards for retraining and fine-tuning cycles to prevent quality drift after model updates.

Collaboration & Leadership

Partner with client AI engineers to validate solutions built using TensorFlow, PyTorch, LangChain, LangGraph, and LlamaIndex.

Define quality KPIs and acceptance criteria for AI features, and report quality posture to engineering and product leadership.

Mentor QA engineers on AI evaluation methodologies, ML fundamentals, and modern test automation practices.

Champion responsible AI practices, including safety, transparency, explain ability, and compliance with evolving AI governance standards.

Required Qualifications

10+ years of professional experience in Quality Engineering and Test Automation, validating complex enterprise applications.

Proficient in validating AI/ML systems, including Generative AI and LLM-based applications.

Strong proficiency in Python and experience building automation frameworks from the ground up.

Practical experience with prompt validation, agentic workflow testing, and AI model evaluation.

Working knowledge of evaluation metrics: BLEU, ROUGE, embedding similarity, precision, recall, F1-score, and human-evaluation methodologies.

Experience with AI/ML frameworks and ecosystems: TensorFlow, PyTorch, LangChain, LangGraph, and LlamaIndex.

Solid understanding of data validation techniques: EDA, schema validation, cross-validation, and statistical analysis.

Experience integrating automated testing into CI/CD pipelines (e.g., GitHub Actions, Jenkins, GitLab CI, Azure DevOps).

Familiarity with bias detection, fairness assessment, and AI safety evaluation techniques.

Preferred Qualifications

Experience with vector databases, retrieval-augmented generation (RAG), and embedding pipelines.

Background in MLOps tooling such as MLflow, Weights & Biases, or similar experiment tracking platforms.

Exposure to LLM observability and evaluation tools (e.g., LangSmith, Ragas, DeepEval, TruLens).

Familiarity with cloud AI services on AWS, Azure, or GCP (Bedrock, Azure OpenAI, Vertex AI).

Knowledge of AI governance frameworks, model cards, and emerging AI regulatory standards.

Bachelor's or Master's degree in Computer Science, Data Science, or a related technical field.