Senior QA Lead, Agentic AI Workflows

Innodata; San Jose (CA); On-site; USD 103,000 - 117,000; Full time

Job summary

Innodata is seeking a Quality Lead to stand up an onsite team in San Jose for evaluating complex agentic AI workflows. You’ll own the audit system, calibrate reviewer output, and drive improvements with the customer.

The role is a senior individual contributor, with responsibility for defining scoring standards, maintaining rubrics, and training reviewers. You will report insights to the Engagement Manager and customer while ensuring privacy and security practices onsite.

Qualifications

• Bachelor's degree or equivalent practical experience.
• 4+ years in quality assurance or senior review roles in annotation, evaluation, trust and safety, or a judgment-intensive domain.
• Experience owning a quality function: designed the audit, not just executed.
• Significant experience with AI/ML evaluation: annotation, red-teaming, RLHF, model/agent evaluation.
• Hands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments.
• Ability to run calibration with peers and resolve disagreements.
• Strong written communication to document scoring standards clearly.
• Proficiency with spreadsheets and dashboarding tools; some Python/SQL for data analysis.
• Experience training/reviewers into rubric-based work.

Responsibilities

• Own the quality system for the engagement: audit design, sampling strategy, scoring standards, and reporting.
• Build the system with the customer's quality leads and drive concrete improvements.
• Re-score a sample of reviewer output as a second pass and identify error patterns.
• Run calibration sessions and document the reasoning for future consistency.
• Maintain rubric health and drive revisions with the customer.
• Train and onboard new reviewers, including ramp criteria and readiness checks.
• Provide evidence behind performance conversations to the Engagement Manager.
• Report quality trends to the Engagement Manager and the customer.
• Deputize for the Engagement Manager on delivery operations during absences.
• Maintain information security, privacy, and facility access practices for onsite environment.

Skills

QA governance

AI evaluation

Calibration sessions

Documentation

Spreadsheet basics

Python/SQL basics

Education

Bachelor's degree or equivalent practical experience

Job description

Innodata is seeking a Quality Lead to stand up an onsite team in San Jose for evaluating complex agentic AI workflows. You’ll own the audit system, calibrate reviewer output, and drive improvements with the customer.

The role is a senior individual contributor, with responsibility for defining scoring standards, maintaining rubrics, and training reviewers. You will report insights to the Engagement Manager and customer while ensuring privacy and security practices onsite.