Senior Quality Lead — Agentic AI Workflow Evaluation

Innodata Inc.; San Jose (CA); On-site; USD 103,000 - 117,000; Full time

Job summary

Innodata Inc. is standing up an onsite team to evaluate complex real-world agentic AI workflows for a frontier customer. The Quality Lead is a senior individual contributor responsible for audit design, calibration, and governance.

You will own the quality system, train reviewers, run calibration sessions, and report trends to the Engagement Manager and customer; this role is on-site in the San Jose area.

Qualifications

• Bachelor's degree or equivalent practical experience.
• 4+ years in quality assurance, quality management, or senior review in annotation/evaluation.
• Experience owning a quality function and designing audits.
• Hands-on AI/ML evaluation experience (RLHF, trust/safety).
• Experience with agentic systems and sandboxed environments.
• Strong written communication and clear scoring standards.
• Proficiency with spreadsheets; some Python/SQL ability.

Responsibilities

• Own the quality system for the engagement: audit design, sampling, scoring, and reporting.
• Build and calibrate the system with the customer's quality leads.
• Re-score reviewer outputs; identify error patterns.
• Run calibration sessions and document resolutions.
• Maintain rubric health and drive revisions with the customer.
• Train and onboard reviewers with ramp criteria.
• Provide evidence on performance trends to the Engagement Manager.
• Depooize for the Engagement Manager during absences.
• Ensure information security, privacy, and facility access per onsite setup.

Skills

Quality Assurance

Quality Management

AI/ML Evaluation

Calibration

Python/SQL basics

Education

Bachelor's degree or equivalent practical experience

Tools

SQL

Python

Dashboard tools

Job description

Innodata Inc. is standing up an onsite team to evaluate complex real-world agentic AI workflows for a frontier customer. The Quality Lead is a senior individual contributor responsible for audit design, calibration, and governance.

You will own the quality system, train reviewers, run calibration sessions, and report trends to the Engagement Manager and customer; this role is on-site in the San Jose area.