Senior AI Evaluation Specialist: Safeguard & Optimize LLMs

The Big Jobsite; Aguascalientes; Remote; MXN 900,000 - 1,300,000; Full time

Job summary

Capgemini Mexico is seeking an experienced AI/ML specialist to evaluate Gemini-based AI agents and ensure responses are accurate, safe, and aligned with business goals. You will design real-world tests, analyze outputs for hallucinations and errors, and provide improvements.

You will collaborate with AI Engineers, Product Owners, and Business teams to optimize prompts, workflows, and tool integrations while safeguarding responsible AI and governance.

Qualifications

• 3+ years in AI/ML, Data Science, NLP, Conversational AI, QA Testing, Prompt Engineering, or related technical fields.
• 1+ year working with Generative AI, LLMs (Gemini, GPT, Claude, Llama), AI chatbots, RAG solutions, or AI agent evaluation.
• Experience evaluating AI responses for accuracy, relevance, groundedness, consistency, and overall business value.
• Prompt engineering and testing expertise, capable of designing evaluation scenarios, edge cases, and identifying opportunities to improve performance.
• Knowledge of AI agents, RAG, and tool-integrated workflows, including how AI systems retrieve information, reason through tasks, and interact with external tools and data sources.
• Commitment to Responsible AI and quality excellence, ensuring AI solutions meet safety, governance, authorization, and user trust requirements before production deployment.
• Fluent English level, good communication skills

Responsibilities

• Evaluate and benchmark Gemini-based AI agents to ensure responses are accurate, relevant, grounded, safe, and aligned with business objectives.
• Design and execute real-world test scenarios, including happy paths, edge cases, and adversarial prompts to identify performance gaps and risks.
• Analyze AI outputs to detect hallucinations, reasoning errors, retrieval issues, and safety concerns, then provide actionable improvement recommendations.
• Work closely with AI Engineers, Prompt Engineers, Product Owners, and Business teams to improve prompts, workflows, tool integrations, and overall agent performance.
• Own the quality and release-readiness assessment of AI solutions by defining evaluation metrics, tracking results, and supporting continuous optimization throughout pilot and production phases.

Skills

AI/ML experience

Prompt engineering

Evaluation skills

RAG workflows

Responsible AI

English fluency

Job description

Capgemini Mexico is seeking an experienced AI/ML specialist to evaluate Gemini-based AI agents and ensure responses are accurate, safe, and aligned with business goals. You will design real-world tests, analyze outputs for hallucinations and errors, and provide improvements.

You will collaborate with AI Engineers, Product Owners, and Business teams to optimize prompts, workflows, and tool integrations while safeguarding responsible AI and governance.