AI QA Engineer (GenAI & ML Systems)

Daimler AG (Canada); Bengaluru; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Mercedes-Benz Research and Development India Private Limited in Bangalore is hiring an AI QA Engineer to own the quality, reliability and safety of GenAI and ML-driven engineering applications. You will design tests for prompts, RAG pipelines and agentic workflows, and set the release bar for AI features used across Powertrain Engineering.

You will build evaluation frameworks, golden datasets, and automated pipelines, run adversarial tests, validate retrieval quality, and define quality gates.

Qualifications

• 4-8 years in QA/SDET with ML or GenAI testing experience.
• Strong Python and test automation skills (pytest).
• Hands-on with LLM evaluation tools and prompt engineering concepts.
• Familiar with LLM behavior, RAG architectures and embeddings.
• Statistical thinking for sampling and metric design.
• CI/CD familiarity with GitHub Actions or Azure DevOps; cloud basics.

Responsibilities

• Design evaluation frameworks for LLM, RAG and agentic systems.
• Build golden datasets, regression suites and automated eval pipelines.
• Run red-teaming and adversarial tests for prompts and data handling.
• Validate retrieval quality, tool-calling and multi-step agent behavior.
• Define quality gates and release criteria for AI features.
• Monitor production drift and lead root-cause analysis with AI engineers.
• Report quality metrics and risks to engineering leadership.

Skills

QA/SDET experience

Python

Test automation

LLM evaluation tools

LLM behavior knowledge

Statistics

CI/CD

Git

Azure fundamentals

Tools

pytest

Ragas

DeepEval

promptfoo

LangSmith

GitHub Actions

Azure DevOps

Azure

Job description

Description

We hiring an AI QA Engineer to own the quality, reliability and safety of our GenAI and ML-driven engineering applications. You will define how we test non-deterministic systems, from prompts and RAG pipelines to agentic workflows, and set the release bar for AI features used across Powertrain Engineering.

Key Responsibilities

• Design evaluation frameworks for LLM, RAG and agentic systems: accuracy, hallucination, consistency, latency and cost
• Build golden datasets, regression suites and automated eval pipelines integrated into CI/CD
• Run red-teaming and adversarial tests: prompt injection, jailbreaks, data leakage, bias and toxicity
• Validate retrieval quality, tool-calling and multi-step agent behavior
• Define quality gates and release criteria for AI features
• Monitor production drift and failure modes; lead root-cause analysis with AI engineers
• Report quality metrics and risks to engineering leadership

Qualifications

Required Qualifications

• 4-8 years in QA/SDET, including 1-2+ years testing ML or GenAI systems
• Strong Python and test automation (pytest, API testing)
• Hands-on with LLM evaluation tools such as Ragas, DeepEval, promptfoo or LangSmith
• Working knowledge of LLM behavior, RAG architectures, embeddings and prompt engineering
• Statistical thinking: sampling, metric design, confidence, and the limits of LLM-as-judge
• CI/CD (GitHub Actions or Azure DevOps), Git and cloud fundamentals (Azure preferred)

Preferred Qualifications

• Experience with agent frameworks and MCP/tool integrations
• Observability tooling for tracing and drift monitoring
• Exposure to engineering or automotive domains

Who Qualifies

Skeptical by default, precise, comfortable with non-deterministic outputs, a clear writer, and a close collaborator to AI engineers.

Experience: 4-8 years

Employment: Full-time

Organization

Mercedes-Benz Research and Development India Private Limited

Primary Location

India-Karnataka-Bangalore

Work Locations

Mercedes-Benz Research & Development India, Embassy Crest 'Embassy Crest' Plot No 5, EPIP Zone, Phase 1, 560 066