Functional AI Tester - GenAI

Michelin Oy; Pune District; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Michelin Oy seeks a Functional AI Tester focused on GenAI applications. The role emphasizes Python-based test automation, GenAI evaluation, and ETL/data quality validation.

You will design end-to-end test strategies and work with AI/data engineers and product teams to define measurable acceptance criteria for high-quality AI features. The position involves testing GenAI features, RAG, conversational AI, guardrail tests, and data pipeline validation, with a strong hands-on collaboration focus.

Qualifications

• 5+ years in software QA, including test strategy, automation, and defect management.
• 2+ years testing AI/ML or GenAI features with hands-on evaluation design.
• 4+ years testing ETL/data pipelines and data quality.

Responsibilities

• Define risk-based test strategies and detailed test plans for GenAI features.
• Build and maintain automated test suites using Python (PyTest, requests).
• Develop evaluation harnesses for factuality, coherence, safety, and bias.
• Verify retrieval accuracy in RAG and perform adversarial tests.
• Test REST endpoints and ETL/data pipelines for quality and schema compliance.
• Share results and provide remediation recommendations.

Skills

Python

PyTest

Requests

API testing

GenAI evaluation

RAG testing

ETL testing

Data quality testing

Data profiling

Tools

Pydantic

Job description

Functional AI Tester - GenAI- - - - - - - - - - - -We are seeking a Quality Assurance (QA) Engineer focused on testing Generative AI (GenAI) applications with a strong emphasis on Python-based test automation, GenAI evaluation, and ETL/data quality validation. You will design and execute end-to-end test strategies that ensure our AI solutions are accurate, reliable, safe, and compliant.**About the Role**You will be involved in QA for GenAI features including Retrieval-Augmented Generation (RAG), conversational AI and Agentic evaluations. The role centers on:* Systematic GenAI evaluation (qualitative and quantitative metrics)* ETL and data quality testing for the data flows that feed AI systems* Python-driven automated testingThis position is hands-on and collaborative, partnering with AI engineers, data engineers, and product teams to define measurable acceptance criteria and ship high-quality AI features.**Key Responsibilities*** Test strategy and planning + Define risk-based test strategies and detailed test plans for GenAI features. + Establish clear acceptance criteria with stakeholders for functional, safety, and data quality aspects.* Python test automation + Build and maintain automated test suites using Python (e.g., PyTest, requests). + Implement reusable utilities for prompt/response validation, dataset management, and result scoring. + Create regression baselines and golden test sets to detect quality drift.* GenAI evaluation + Develop evaluation harnesses covering factuality, coherence, helpfulness, safety, bias, and toxicity etc. + Design prompt suites, scenario-based tests, and golden datasets for reproducible measurements. + Implement guardrail tests including prompt-injection resilience, unsafe content detection, and PII redaction checks. + Track quality metrics over time.* RAG and semantic retrieval testing + Verify alignment between retrieved sources and generated answers. + Verify adversarial tests. + Measure retrieval relevance, precision/recall, grounding quality, and hallucination reduction.* API and application testing + Test REST endpoints supporting GenAI features (request/response contracts, error handling, timeouts).* ETL and data quality validation + Test ingestion and transformation logic; validate schema, constraints, and field-level rules. + Implement data profiling, reconciliation between sources and targets, and lineage checks. + Verify data privacy controls, masking, and retention policies across pipelines.* Non-functional testing + Performance and load testing focused on latency, throughput, concurrency, and rate limits for LLM calls. + Cost-aware testing (token usage, caching effectiveness) and timeout/retry behavior validation. + Reliability and resilience checks including error recovery and fallback behavior.* Share results and insights; recommend remediation and preventive actions.**Required Qualifications*** Experience + 5+ years in software QA, including test strategy, automation, and defect management. + 2+ years testing AI/ML or GenAI features, with hands-on evaluation design. + 4+ years testing ETL/data pipelines and data quality.* Technical skills + Python: Strong proficiency building automated tests and tooling (PyTest, requests, pydantic or similar). + API testing: REST contract testing, schema validation, negative testing. + GenAI evaluation: crafting prompt suites, golden datasets, rubric-based scoring, and automated evaluation pipelines. + RAG testing: retrieval relevance, grounding validation, chunking/indexing verification, and embedding checks. + ETL/data quality: schema and constraint validation, reconciliation, lineage awareness, data profiling.* Quality and governance + Understanding of LLM limitations and methods to detect/reduce hallucinations. + Safety and compliance testing including PII handling and prompt-injection resilience. + Strong analytical and debugging skills across services and data flows.* Soft skills + Excellent written and verbal communication; ability to translate quality goals into measurable criteria. + Collaboration with AI engineers, data engineers, and product stakeholders. + Organized, detail-oriented, and outcomes-focused.**Nice to Have*** Experience with evaluation frameworks or tooling for LLMs and RAG quality measurement.Experience creating synthetic datasets to stress specific behaviors.