AI Evaluation & Quality Lead for LLMs

Capgemini; Aguascalientes; Remote; MXN 540,000 - 900,000; Full time

Job summary

Capgemini in Mexico offers an AI/ML quality assurance role focused on evaluating Gemini-based AI agents for accuracy, grounding, safety, and business value. The role blends technical testing with practical business impact and requires hands-on design of test scenarios, edge cases, and adversarial prompts.

You will work with AI Engineers, Prompt Engineers, Product Owners, and Business teams to improve prompts, workflows, and tool integrations, ensuring release-readiness and responsible AI

Qualifications

• 3+ years in AI/ML, Data Science, NLP, Conversational AI, QA Testing, or related technical fields.
• 1+ year working with Generative AI, LLMs, AI chatbots, RAG solutions, or AI agent evaluation.
• Experience evaluating AI responses for accuracy, relevance, grounding, consistency and business value.
• Prompt engineering and testing expertise, capable of designing evaluation scenarios and edge cases.
• Knowledge of AI agents, RAG and tool-integrated workflows and how AI retrieves information.

Responsibilities

• Evaluate and benchmark Gemini-based AI agents to ensure accuracy, grounding, safety and business alignment.
• Design and execute real-world test scenarios including happy paths, edge cases and adversarial prompts.
• Analyze AI outputs to detect hallucinations, reasoning errors, retrieval issues, and safety concerns; propose improvements.
• Collaborate with AI Engineers, Prompt Engineers, Product Owners and Business teams to improve prompts, workflows and tool integrations.
• Own quality and release-readiness assessments of AI solutions across pilot and production phases.

Skills

AI/ML

NLP

QA Testing

Prompt Engineering

Generative AI

LLMs

Tool integrations

Responsible AI

English fluent

Tools

Gemini

GPT

Claude

Llama

RAG

External tools

Job description

Capgemini in Mexico offers an AI/ML quality assurance role focused on evaluating Gemini-based AI agents for accuracy, grounding, safety, and business value. The role blends technical testing with practical business impact and requires hands-on design of test scenarios, edge cases, and adversarial prompts.

You will work with AI Engineers, Prompt Engineers, Product Owners, and Business teams to improve prompts, workflows, and tool integrations, ensuring release-readiness and responsible AI