AI Agent Evaluation Engineer

Letitbex AI; Telangana; On-site; INR 4,000,000 - 7,500,000; Full time

Job summary

LetitbexAI is seeking an AI Agent Evaluation Engineer to build and run comprehensive evaluation frameworks for AI agents using Google ADK. The role emphasizes Responsible AI, safety evaluations, and robust testing strategies, balancing automation (70%) with manual testing (30%).

The ideal candidate has 8+ years in software QA with AI/ML experience, strong Python skills, and familiarity with GCP Vertex AI and evaluation tools, capable of clear communication of risks and findings.

Qualifications

• 8+ years in Software QA with 2–3 years testing AI/ML systems, conversational agents, or LLMs.
• Direct experience in safety evaluations (red teaming, adversarial testing) and bias/toxicity detection in generative AI models.
• Proven experience developing and running evaluations for LLM-powered applications using libraries like PyTest.

Responsibilities

• Define, develop, and execute robust Agent evaluation frameworks and test strategies with emphasis on Responsible AI and Safety Evaluations.
• Bridge AI development and deployment to ensure safe, ethical, and high-quality agent performance.
• Lead automation efforts (roughly 70%) with supporting manual testing (about 30%).

Skills

QA Experience

Safety Evaluations

LLM Evaluations

Google ADK

Python

GCP Vertex AI

Communication Skills

Evaluation Tools

Tools

PyTest

Job description

LetitbexAI is a fast-growing AI-driven technology company focused on building intelligent, scalable, and enterprise-grade solutions. We work at the intersection of AI, data engineering, cloud, and business transformation, helping organizations unlock real value from artificial intelligence.

Position: AI Agent Evaluation Engineer

Experience: 8+ Years

Notice Period: Can be considered up 15 Days

Job Position (Title) : AI Agent Evaluation Engineer

Experience: 8+ Years

Job Summary ::

We are seeking a highly motivated and technically proficient AI Agent Evaluation Engineer to join our growing AI team. This crucial role will be responsible for defining, developing, and executing robust Agent evaluation frameworks and test strategies, with a significant focus on Responsible AI and Safety Evals, for our agents built using the Google Agent Development Kit (ADK). The ideal candidate will bridge the gap between AI development and reliable deployment, ensuring our agents are safe, ethical, effective, and meet high-quality performance standards. The role will be of 70% Automation and 30% Manual Testing

Required Skills & Qualifications:

• Experience: 8+ years in Software QA, with at least 2-3 years focused on testing or evaluating AI/ML systems, conversational agents, or Large Language Models (LLMs).
• Safety Evals Expertise (Mandatory): Direct experience in designing and executing safety evaluations (red teaming, adversarial testing), bias detection, and measuring toxicity/harmful content in generative AI models.
• Agent/LLM Evals: Proven experience developing and running general evaluations (Evals) for LLM-powered applications knowing libraries like PyTest (Must)
• Google ADK Familiarity (Mandatory): Direct or strong conceptual understanding of the Google Agent Development Kit (ADK) and its components.
• Programming: Strong proficiency in Python is mandatory for script development, data processing, and automation.
• Cloud & MLOps: Familiarity with Google Cloud Platform (GCP) services relevant to AI/ML (e.g., Vertex AI) and integrating testing into MLOps workflows.
• Communication: Excellent analytical, problem-solving, and communication skills to articulate complex agent behaviors, risks, and safety findings clearly.
• Tools and Libraries: Langsmith, DeepEval, Ragas, Giskard, Hugging face.