AI QA Engineer

APEX Analytix, LLC

Dadri

On-site

INR 800,000 - 1,400,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Mentor program
Education assistance
Tuition reimbursement
Certification assistance
Competitive benefits

Job summary

apexanalytix is seeking an AI QA Engineer to drive quality, reliability, and safety of AI solutions for Global 2000 clients. You will shape test strategy for LLM features, agentic workflows, and ML models, plus build regression suites with experts.

You will develop automated evaluation pipelines (DeepEval, Ragas, LangSmith) and perform non-deterministic output testing, including accuracy and bias assessments, collaborating with Data Science and Engineering teams.

Qualifications

  • BS/MS in Computer Science, Data Science, ML/AI, Statistics, or related field.
  • 2–3 years in AI/ML, data science, or AI-focused QA with 1+ year of automation.
  • Experience evaluating GenAI apps (RAG, agentic workflows) with DeepEval/Ragas.
  • Strong Python, PyTest, API testing, and CI/CD integration.
  • Grounding in statistics, sampling, validation and SQL.
  • Experience testing unstructured data (text/images).

Responsibilities

  • Own the test strategy for AI products, including LLM features and ML models.
  • Write and execute test plans, manage defects, and perform exploratory testing.
  • Define acceptance criteria for non-deterministic outputs using human evaluation.
  • Build golden datasets and regression suites reflecting real scenarios.
  • Develop automated evaluation pipelines (DeepEval, Ragas, LangSmith) in CI/CD.
  • Test agentic behavior, tool calling, integrations, and guardrails.
  • Validate data pipelines for quality, bias, and drift.

Skills

Python
PyTest
API testing
LLM evaluation
ML model validation
Statistics concepts
SQL
Communication

Education

BS/MS in CS/DS/AI

Tools

DeepEval
Ragas
promptfoo
LangSmith
LangChain
LangGraph
Hugging Face
Great Expectations
Evidently
Playwright
Selenium
Docker
Kubernetes

Job description

At apexanalytix, we’re lifelong innovators! Since the date of our founding nearly four decades ago we’ve been consistently growing, profitable, and delivering the best procure-to-pay solutions to the world. We’re the perfect balance of established company and start-up. You will find a unique home here.

And you’ll recognize the names of our clients. Most of them are on The Global 2000. They trust us to give them the latest in controls, audit and analytics software every day. Industry analysts consistently rank us as a top supplier management solution, and you’ll be helping build that reputation.

As an AI QA Engineer, you’ll drive the quality, reliability, and safety of the AI solutions we deliver to our Global 2000 clients, including GenAI and agentic applications, RAG pipelines, and ML models used across our products. This role is for an engineer with a solid data science/AI background who understands how models behave and is skilled in LLM evaluation, end-to-end AI testing, and ML model validation, helping us ship production-grade AI with confidence.

The Work
  • Own the test strategy for our AI products, covering LLM features, agentic workflows, RAG pipelines, and ML models.
  • Write and execute test plans, cases, and reports; manage the defect lifecycle; and perform exploratory testing of AI features.
  • Define acceptance criteria and quality metrics for non-deterministic outputs (accuracy, groundedness, hallucination rate, relevance, cost) using LLM-as-a-judge and human evaluation.
  • Build and maintain golden datasets, ground-truth labels, and regression suites with subject-matter experts, reflecting real scenarios and edge cases.
  • Develop automated evaluation pipelines (DeepEval, Ragas, promptfoo, LangSmith) that run in CI/CD on every prompt, model, retrieval, or code change.
  • Test agentic behavior—multi-step planning, tool calling, integrations, error handling, and fallbacks—to verify agents’ complete tasks correctly and stay within guardrails.
  • Validate ML models and data pipelines for data quality, segment performance, bias/fairness, and drift, using statistical methods to compare versions.
  • Perform adversarial and safety testing: prompt injection, jailbreaks, PII leakage, and compliance checks.
  • Monitor production AI quality through observability tooling, triage degradations, lead root-cause analysis, and partner with Data Science, AI, and engineering teams on fixes and release sign-off.
The Must-Haves
  • B.S./M.S. in Computer Science, Data Science, ML/AI, Statistics, or related field.
  • 2–3 years in AI/ML, data science, or AI-focused QA, including 1+ year of hands-on test automation, ideally for production ML systems.
  • Working knowledge of ML model building, evaluation metrics, and LLM behavior (prompting, context, RAG).
  • Experience evaluating GenAI applications (RAG pipelines, agentic workflows, function calling) with DeepEval, Ragas, or similar.
  • Strong Python with PyTest, API testing, and CI/CD integration.
  • Grounding in statistics and experimentation (sampling, significance testing, model A/B comparison), plus SQL and data validation tools such as Great Expectations or Evidently.
  • Experience testing document and image analysis solutions that extract insights from unstructured data.
  • Ability to define pass/fail criteria and rubrics for non-deterministic outputs and translate business requirements into measurable quality standards.
  • Clear written and verbal communication with technical and business stakeholders across time zones.
The Good-to-Haves
  • Synthetic test data generation, including privacy-safe datasets, edge cases, and adversarial scenarios.
  • Docker, Kubernetes, and MLOps tooling for deploying and testing AI applications.
  • Familiarity with AI risk and governance frameworks (NIST AI RMF, ISO/IEC 42001, EU AI Act) and responsible AI practices (OWASP Top 10 for LLMs, bias testing, PII protection).
  • Working knowledge of LangChain, LangGraph, Hugging Face, or similar, plus LLM APIs (Anthropic, OpenAI, open-source models).
  • UI and end-to-end test automation (Playwright, Selenium) for AI-driven interfaces.

Over the years, we’ve discovered that the most effective and successful associates at apexanalytix are people who have a specific combination of values, skills, and behaviors that we call “The apex Way”. Read more about The apex Way - https://www.apexanalytix.com/careers/

At apexanalytix we know that our associates are the reason behind our successes. We truly value you as an associate and part of our professional family. Our goal is to offer the very best benefits possible to you and your loved ones. When it comes to benefits, whether for yourself or your family the most important aspect is choice. And we get that. apexanalytix offers competitive benefits for the countries that we serve, in addition to our BeWell@apex initiative that encourages employees’ growth in six key wellness areas: Emotional, Physical, Community, Financial, Social, and Intelligence.

With resources such as a strong Mentor Program, Internal Training Portal, plus Education, Tuition, and Certification Assistance, we provide tools for our associates to grow and develop.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI QA Engineer
AI QA Engineer

apexanalytix • Dadri

On-site
INR 1,200,000 - 2,200,000
AI Engineer (Agentic AI & Regulatory Reporting)
AI Engineer (Agentic AI & Regulatory Reporting)

Apexon • Bengaluru

On-site
INR 1,500,000 - 2,100,000
Group Health Insurance
Paid Holidays & Earned Leaves
Employee Wellness
AI Infrastructure UI/UX Architect & Engineer Manager
AI Infrastructure UI/UX Architect & Engineer Manager

CO_AFATI Apex Fund Services LLP • Maharashtra

On-site
INR 1,200,000 - 2,400,000
Senior Quality Assurance Engineer
Senior Quality Assurance Engineer

apexanalytix • Dadri

On-site
INR 900,000 - 1,800,000
Mentor Program
Education & Certification Support
Tuition Assistance
VP -Product
VP -Product

apexanalytix • Dadri

Hybrid
INR 17,078,000 - 26,565,000
Mentor program
Internal training portal
Education, tuition & certification aid
+1
VP -Product
VP -Product

apexanalytix • Dadri

Hybrid
INR 17,029,000 - 26,490,000
Quality Assurance Engineer – AI Agentic Applications, AVP
Quality Assurance Engineer – AI Agentic Applications, AVP

Deutsche Bank AG • Maharashtra

On-site
INR 1,500,000 - 2,000,000
Leave policy
Parental leaves (gender neutral)
Childcare assistance
+5
Junior GenAI Full Stack Engineer – Agentic AI & RAG
Junior GenAI Full Stack Engineer – Agentic AI & RAG

Apexon • Ahmedabad District

On-site
INR 1,200,000 - 1,800,000
Group Health Insurance for family of 4
Term Insurance and Accident Insurance
Paid Holidays & Earned Leaves
+1
Vice President, Engineering
Vice President, Engineering

apexanalytix • Dadri

On-site
INR 23,764,000 - 33,270,000
AI/ML Engineer (Gen AI)
AI/ML Engineer (Gen AI)

Apexon • Bengaluru

On-site
INR 1,000,000 - 1,500,000
Group Health Insurance
Term Insurance
Paid Holidays
+1