Onsite Delivery Lead, Agentic AI Evaluations

Innodata; San Jose (CA); On-site; USD 103,000 - 117,000; Full time

Job summary

Innodata in San Jose, California, seeks an Engagement Manager to lead an onsite team evaluating agentic AI workflows. You will shepherd throughput, quality, and timely delivery while serving as the customer's main point of contact with technical leads.

You’ll recruit, onboard, and retain reviewers and partner with QA for calibration and rubric refinement. The role requires 5+ years of management experience, strong communication, and proven ability to own a complex delivery engagement with

Qualifications

• Bachelor's degree or equivalent practical experience.
• 5+ years of professional experience managing a delivery team of 10–20 people.
• Track record as the customer-facing owner of a services or delivery engagement with an enterprise or technology client, including metrics reporting and issue escalation.
• Significant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review.
• Ability to read rubric-scored work critically and form an independent view of whether a score is correct.
• Strong written communication; able to produce reporting and escalation memos that a technical customer will accept without rework.

Responsibilities

• Own end-to-end delivery for the onsite team: throughput, quality, turnaround, and capacity against committed volumes.
• Manage and develop a team of eleven, including hiring, onboarding, performance management, and retention.
• Serve as the primary onsite point of contact for the customer's program and technical leads; run recurring reviews, report on delivery metrics, and resolve issues before they elevate.
• Audit reviewer output directly — sample trajectories, check rubric application, and assess whether scoring rationale would survive customer review.
• Partner with the QA lead on calibration cycles, quality metrics and evaluation, and rubric refinement; arbitrate disagreements that calibration does not resolve.
• Own the escalation path for safety-relevant and ambiguous findings, including judgment on what reaches the customer and how quickly.
• Identify expansion opportunities within the account: adjacent workstreams, new task types, additional capacity, and scope the work with internal delivery and commercial teams.
• Forecast staffing and cost against the engagement's commercial model, and flag variance early
• Maintain information security, privacy, and facility access practices required by the customer's onsite environment

Skills

People management

Delivery leadership

Customer facing

AI/ML evaluation

Rubric calibration

Reporting

Education

Bachelor's degree or equivalent

Job description

Innodata in San Jose, California, seeks an Engagement Manager to lead an onsite team evaluating agentic AI workflows. You will shepherd throughput, quality, and timely delivery while serving as the customer's main point of contact with technical leads.

You’ll recruit, onboard, and retain reviewers and partner with QA for calibration and rubric refinement. The role requires 5+ years of management experience, strong communication, and proven ability to own a complex delivery engagement with