AI Quality Engineer - 17397

Seneca Resources Company, LLC; Vienna (VA); Hybrid; USD 73,012 - 82,656; Part time

Job summary

Seneca Resources seeks an AI Quality Engineer to lead validation, certification, and production readiness for enterprise AI solutions. You will define evaluation methods for AI accuracy, relevance, groundedness, and completeness while building datasets and benchmarks.

You will collaborate with AI engineers, security teams, and product stakeholders to ensure scalable, auditable AI deployments with robust governance and monitoring practices.

Qualifications

• Bachelor's degree or higher in a technical field.
• 5+ years in Software Quality Engineering, test architecture, or related roles.
• 2+ years with Generative AI, LLMs, AI Agents, or RAG.
• Strong AI evaluation techniques: hallucination, groundedness, accuracy, and quality metrics.
• Experience with Azure cloud services and AI tooling.
• Familiarity with CI/CD, DevSecOps, and enterprise SDLC practices.

Responsibilities

• Develop AI validation, certification, and production readiness standards for enterprise AI solutions.
• Design evaluation frameworks: AI accuracy, relevance, groundedness, completeness, hallucination detection.
• Build AI validation datasets, benchmarks, regression suites, and golden datasets.
• Validate RAG solutions using Azure AI Search, LangChain, LangGraph, and LangSmith.
• Review AI architectures deployed on Azure services and ensure security and governance controls.
• Produce certification reports, scorecards, dashboards, and executive summaries for governance.
• Lead quality gates and certification criteria for enterprise AI deployments and platform initiatives.
• Drive continuous improvement of AI testing strategies and quality engineering practices.

Skills

Generative AI

LLMs

AI Agents

RAG solutions

Azure AI Foundry

LangChain

LangSmith

AI Evaluation

Education

Bachelor's degree in Computer Science or related field

Tools

Azure AI Foundry

Azure AI Search

LangChain

LangGraph

Cosmos DB

Azure Databricks

Job description

Position Title: AI Quality Engineer
Location: Vienna, VA / Remote
Clearance Requirements: None
Position Status: Contract
Pay Rate: $53 - $60 per hour

Position Description: We are seeking an experienced AI Quality Engineer to lead the validation, certification, and production readiness of enterprise Generative AI and AI-powered automation solutions. This is a highly technical engineering role focused on ensuring AI systems are accurate, reliable, secure, explainable, compliant, and ready for enterprise production deployment.

This is not a traditional QA or manual testing position. The ideal candidate will serve as an independent quality authority responsible for evaluating Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) applications, AI agents, orchestration frameworks, developer platforms, and AI-enabled SDLC solutions.

Working alongside AI Engineers, Platform Engineers, Architects, Security, Risk, DevOps, and Product teams, you will develop repeatable validation frameworks, AI evaluation methodologies, production readiness standards, and governance processes that enable the successful deployment of enterprise AI solutions.

This position is ideal for engineers passionate about AI Quality Engineering, Responsible AI, AI Governance, Platform Engineering, DevEx, Azure AI, and enterprise-scale automation.

Key Responsibilities:

• Key Responsibilities:
• Develop and implement AI validation, certification, and production readiness standards for enterprise AI solutions.
• Design evaluation frameworks to measure:
  • AI accuracy
  • Response relevance
  • Groundedness
  • Completeness
  • Hallucination detection
  • Retrieval effectiveness
  • Recommendation quality
  • User satisfaction
• Build and maintain AI validation datasets, benchmark scenarios, regression suites, and golden datasets using tools such as LangSmith and Azure AI Foundry.
• Validate RAG (Retrieval-Augmented Generation) solutions utilizing Azure AI Search, LangChain, LangGraph, Azure AI Foundry, and enterprise knowledge repositories.
• Evaluate AI agent workflows, orchestration pipelines, prompt execution, tool integrations, guardrails, human-in-the-loop processes, and MCP integrations.
• Assess AI security, governance, auditability, identity management, and compliance controls including Entra ID, RBAC, Managed Identities, Key Vault, and data protection requirements.
• Develop production readiness checklists covering observability, monitoring, resiliency, logging, supportability, recoverability, and operational excellence.
• Analyze AI telemetry, LangSmith traces, execution logs, and evaluation metrics to identify quality issues and optimization opportunities.
• Partner with engineering teams to resolve AI quality, security, and performance concerns before production deployment.
• Produce AI certification reports, quality scorecards, dashboards, and executive summaries for governance reviews.
• Establish independent quality gates and certification criteria for enterprise AI deployments.
• Lead validation and production readiness reviews for Internal Developer Portal (IDP), Developer Experience (DevEx), self-service engineering workflows, and platform automation initiatives.
• Drive continuous improvement of AI testing strategies, evaluation methodologies, and quality engineering practices.

Key Responsibilities:

• Key Responsibilities:
• Develop and implement AI validation, certification, and production readiness standards for enterprise AI solutions.
• Design evaluation frameworks to measure:
  • AI accuracy
  • Response relevance
  • Groundedness
  • Completeness
  • Hallucination detection
  • Retrieval effectiveness
  • Recommendation quality
  • User satisfaction
• Build and maintain AI validation datasets, benchmark scenarios, regression suites, and golden datasets using tools such as LangSmith and Azure AI Foundry.
• Validate RAG (Retrieval-Augmented Generation) solutions utilizing Azure AI Search, LangChain, LangGraph, Azure AI Foundry, and enterprise knowledge repositories.
• Review AI solution architectures deployed across Azure cloud services including:
  • Azure Container Apps
  • Azure Functions
  • Azure Databricks
  • Azure SQL
  • Cosmos DB
• Evaluate AI agent workflows, orchestration pipelines, prompt execution, tool integrations, guardrails, human-in-the-loop processes, and MCP integrations.
• Assess AI security, governance, auditability, identity management, and compliance controls including Entra ID, RBAC, Managed Identities, Key Vault, and data protection requirements.
• Develop production readiness checklists covering observability, monitoring, resiliency, logging, supportability, recoverability, and operational excellence.
• Analyze AI telemetry, LangSmith traces, execution logs, and evaluation metrics to identify quality issues and optimization opportunities.
• Partner with engineering teams to resolve AI quality, security, and performance concerns before production deployment.
• Produce AI certification reports, quality scorecards, dashboards, and executive summaries for governance reviews.
• Establish independent quality gates and certification criteria for enterprise AI deployments.
• Lead validation and production readiness reviews for Internal Developer Portal (IDP), Developer Experience (DevEx), self-service engineering workflows, and platform automation initiatives.
• Drive continuous improvement of AI testing strategies, evaluation methodologies, and quality engineering practices.

Required Skills/Education

• Bachelor's degree in Computer Science, Software Engineering, Artificial Intelligence, Data Science, Information Systems, or a related technical discipline.
• 5+ years of experience in Software Quality Engineering, Test Architecture, Software Development, Platform Engineering, AI Engineering, Machine Learning Engineering, or related technical roles.
• 2+ years of hands-on experience with Generative AI, Large Language Models (LLMs), AI Agents, or Retrieval-Augmented Generation (RAG) solutions.
• Strong understanding of AI evaluation techniques including:
  • Hallucination detection
  • Groundedness validation
  • Accuracy testing
  • AI quality metrics
  • Model evaluation
• Experience with one or more of the following technologies:
  • Azure AI Foundry
  • Azure OpenAI
  • LangChain
  • LangGraph
  • LangSmith
  • AI Agent frameworks
  • RAG architectures
• Experience with Azure cloud technologies including:
  • Azure AI Search
  • Azure Container Apps
  • Azure Functions
  • Cosmos DB
  • Azure SQL
  • Azure Databricks
  • Azure Key Vault
• Experience supporting Platform Engineering, Internal Developer Portals (IDP), DevEx platforms, DevOps, or CI/CD environments.
• Strong knowledge of automated testing frameworks, regression testing, AI validation methodologies, and quality certification processes.
• Experience with APIs, microservices, distributed systems, and cloud-native architectures.
• Familiarity with DevSecOps, CI/CD pipelines, observability, monitoring, and enterprise SDLC practices.
• Excellent analytical, troubleshooting, documentation, and stakeholder communication skills.
• Ability to work independently while providing objective, data-driven quality assessments.

Preferred Qualifications

• Experience validating enterprise AI agents or multi-agent systems.
• Background in Platform Engineering, Developer Experience (DevEx), Site Reliability Engineering (SRE), or DevOps.
• Experience with AI observability and evaluation platforms such as LangSmith.
• Knowledge of Azure AI Search, vector databases, semantic search, embeddings, and enterprise knowledge retrieval.
• Experience implementing Responsible AI, AI Governance, AI Risk Management, or AI Compliance frameworks.
• Experience in highly regulated industries such as financial services, banking, healthcare, or insurance.
• Familiarity with Azure DevOps, GitHub, GitHub MCP, Azure DevOps MCP, and enterprise SDLC tooling.
• Experience with performance engineering, resiliency testing, chaos engineering, and production readiness reviews.
• Knowledge of Entra ID, RBAC, Managed Identities, Azure Key Vault, and identity governance.
• Experience creating executive dashboards, KPIs, quality scorecards, and AI performance reporting.

About Seneca Resources

At Seneca Resources, we are more than just a staffing and consulting firm, we are a trusted career partner. With offices across the U.S. and clients ranging from Fortune 500 companies to government organizations, we provide opportunities that help professionals grow their careers while making an impact.

When you work with Seneca, you’re choosing a company that invests in your success, celebrates your achievements, and connects you to meaningful work with leading organizations nationwide. We take the time to understand your goals and match you with roles that align with your skills and career path. Our consultants and contractors enjoy competitive pay, comprehensive health, dental, and vision coverage, 401(k) retirement plans, and the support of a dedicated team who will advocate for you every step of the way.

Seneca Resources is proud to be an Equal Opportunity Employer, committed to fostering a diverse and inclusive workplace where all qualified individuals are encouraged to apply.