AI Evaluations Team Lead

FNZ; Pune District; On-site; INR 9,541,985 - 12,404,580; Full time

Job summary

FNZ is seeking a Senior AI Evaluations Lead in Pune, India, to oversee AI solutions' safety and compliance within products. You will define strategies, establish mandatory standards, and represent the evaluation function in governance.

Ideal candidates will have 7-12 years of experience in AI/ML, a strong programming background, and proven leadership. This role requires design and execution of complex evaluations for high-risk AI agents, ensuring rigorous performance standards.

Qualifications

• 7-12 years in AI/ML engineering, quality assurance, or security testing with 3+ years in evaluation/red-teaming.
• Deep understanding of LLM-based agents and agentic AI systems.
• Strong programming background with experience in evaluation solutions.

Responsibilities

• Define and evolve AI evaluation strategy aligned to FNZ's framework.
• Establish evaluation standards and scoring criteria.
• Represent evaluations function in AI Governance Committee.
• Build and manage a team of evaluation specialists.
• Design evaluations for high-risk AI agents.

Skills

AI/ML engineering

Quality assurance

Security testing

Evaluation/red-teaming

Programming

Evaluation methodologies

Compliance awareness

Leadership

Job description

Location: Pune, India

Seniority: Senior (7-12 years)

Purpose: Lead FNZ's unified AI evaluations and red teaming capability, ensuring all AI agent solutions meet rigorous safety, performance, and compliance standards before production deployment.

Key Responsibilities

• Define and evolve AI evaluation strategy aligned to FNZ's six‑pillar framework (Task Performance, Safety & Compliance, Efficiency, Groundedness & Reasoning, Robustness, Suitability)
• Establish evaluation standards, methodologies, and scoring criteria integrated into FNZ's SDLC as mandatory release gates
• Represent evaluations function in AI Governance Committee, providing risk assessments and release recommendations
• Build, mentor, and manage a team of 4 evaluation specialists across generalist and specialist roles
• Design and execute complex evaluations for high‑risk AI agents; lead red‑team exercises for critical deployments
• Communicate evaluation findings to technical and non-technical stakeholders; influence product roadmaps

Skills and Experience

• 7‑12 years in AI/ML engineering, quality assurance, or security testing with 3+ years in evaluation/red‑teaming
• Deep understanding of LLM‑based agents, RAG architectures, and agentic AI systems (not model training)
• Strong programming background with real‑world experience developing evaluation solutions for agents
• Proven ability to design evaluation methodologies for probabilistic AI systems
• Experience in regulated industries (financial services, healthcare) with compliance awareness
• Leadership experience managing cross‑functional teams and influencing stakeholders