AI Quality Engineer for Production Readiness

Seneca Resources Company, LLC; Vienna (VA); Hybrid; USD 73,012 - 82,656; Part time

Job summary

Seneca Resources seeks an AI Quality Engineer to lead validation, certification, and production readiness for enterprise AI solutions. You will define evaluation methods for AI accuracy, relevance, groundedness, and completeness while building datasets and benchmarks.

You will collaborate with AI engineers, security teams, and product stakeholders to ensure scalable, auditable AI deployments with robust governance and monitoring practices.

Qualifications

• Bachelor's degree or higher in a technical field.
• 5+ years in Software Quality Engineering, test architecture, or related roles.
• 2+ years with Generative AI, LLMs, AI Agents, or RAG.
• Strong AI evaluation techniques: hallucination, groundedness, accuracy, and quality metrics.
• Experience with Azure cloud services and AI tooling.
• Familiarity with CI/CD, DevSecOps, and enterprise SDLC practices.

Responsibilities

• Develop AI validation, certification, and production readiness standards for enterprise AI solutions.
• Design evaluation frameworks: AI accuracy, relevance, groundedness, completeness, hallucination detection.
• Build AI validation datasets, benchmarks, regression suites, and golden datasets.
• Validate RAG solutions using Azure AI Search, LangChain, LangGraph, and LangSmith.
• Review AI architectures deployed on Azure services and ensure security and governance controls.
• Produce certification reports, scorecards, dashboards, and executive summaries for governance.
• Lead quality gates and certification criteria for enterprise AI deployments and platform initiatives.
• Drive continuous improvement of AI testing strategies and quality engineering practices.

Skills

Generative AI

LLMs

AI Agents

RAG solutions

Azure AI Foundry

LangChain

LangSmith

AI Evaluation

Education

Bachelor's degree in Computer Science or related field

Tools

Azure AI Foundry

Azure AI Search

LangChain

LangGraph

Cosmos DB

Azure Databricks

Job description

Seneca Resources seeks an AI Quality Engineer to lead validation, certification, and production readiness for enterprise AI solutions. You will define evaluation methods for AI accuracy, relevance, groundedness, and completeness while building datasets and benchmarks.

You will collaborate with AI engineers, security teams, and product stakeholders to ensure scalable, auditable AI deployments with robust governance and monitoring practices.