AI QA Engineer (GenAI & ML Systems)

Selectrucks; Bengaluru; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Mercedes-Benz Research and Development India Private Limited in Bengaluru seeks an AI QA Engineer to own quality, reliability and safety of GenAI and ML-powered applications across Powertrain Engineering.

You will design evaluation frameworks, build datasets, perform red-teaming, and define release criteria for AI features, collaborating with AI engineers and engineering leadership.

Qualifications

• 4-8 years in QA/SDET, including 1-2+ years testing ML or GenAI systems.
• Strong Python and test automation (pytest, API testing).
• Hands-on with LLM evaluation tools such as Ragas, DeepEval, promptfoo or LangSmith.
• Working knowledge of LLM behavior, RAG architectures, embeddings and prompt engineering.
• Statistical thinking: sampling, metric design, confidence, and the limits of LLM-as-judge.
• CI/CD (GitHub Actions or Azure DevOps), Git and cloud fundamentals (Azure preferred).

Responsibilities

• Design evaluation frameworks for LLM, RAG and agentic systems: accuracy, hallucination, consistency, latency and cost
• Build golden datasets, regression suites and automated eval pipelines integrated into CI/CD
• Run red-teaming and adversarial tests: prompt injection, jailbreaks, data leakage, bias and toxicity
• Validate retrieval quality, tool-calling and multi-step agent behavior
• Define quality gates and release criteria for AI features
• Monitor production drift and failure modes; lead root-cause analysis with AI engineers
• Report quality metrics and risks to engineering leadership

Skills

Python

Test automation

LLM evaluation

CI/CD

Cloud fundamentals

Tools

Ragas

DeepEval

promptfoo

LangSmith

GitHub Actions

Azure DevOps

Job description

Description

We hiring an AI QA Engineer to own the quality, reliability and safety of our GenAI and ML-driven engineering applications. You will define how we test non-deterministic systems, from prompts and RAG pipelines to agentic workflows, and set the release bar for AI features used across Powertrain Engineering.

Key Responsibilities

• Design evaluation frameworks for LLM, RAG and agentic systems: accuracy, hallucination, consistency, latency and cost
• Build golden datasets, regression suites and automated eval pipelines integrated into CI/CD
• Run red-teaming and adversarial tests: prompt injection, jailbreaks, data leakage, bias and toxicity
• Validate retrieval quality, tool-calling and multi-step agent behavior
• Define quality gates and release criteria for AI features
• Monitor production drift and failure modes; lead root-cause analysis with AI engineers
• Report quality metrics and risks to engineering leadership

Qualifications

Required Qualifications

• 4-8 years in QA/SDET, including 1-2+ years testing ML or GenAI systems
• Strong Python and test automation (pytest, API testing)
• Hands-on with LLM evaluation tools such as Ragas, DeepEval, promptfoo or LangSmith
• Working knowledge of LLM behavior, RAG architectures, embeddings and prompt engineering
• Statistical thinking: sampling, metric design, confidence, and the limits of LLM-as-judge
• CI/CD (GitHub Actions or Azure DevOps), Git and cloud fundamentals (Azure preferred)

Preferred Qualifications

• Experience with agent frameworks and MCP/tool integrations
• Observability tooling for tracing and drift monitoring
• Exposure to engineering or automotive domains

Who Qualifies

Skeptical by default, precise, comfortable with non-deterministic outputs, a clear writer, and a close collaborator to AI engineers.

Experience

4-8 years

Employment

Full-time

Organization

Mercedes-Benz Research and Development India Private Limited

Primary Location

India-Karnataka-Bangalore

Work Locations

Mercedes-Benz Research & Development India, Embassy Crest 'Embassy Crest' Plot No 5, EPIP Zone, Phase 1, 560 066