QA / Evaluation Lead

Innodata; McLean (VA); On-site; USD 61,992 - 68,880; Full time

Job summary

Innodata in McLean, VA, is seeking a QA/Evaluation Lead to own quality across the platform. You will design the evaluation framework, establish IAA metrics, and partner with the Delivery Owner and engineering leads to make quality measurable at each milestone.

This role requires rigorous, evidence-backed thinking about evaluation, experience with Cohen's kappa, Fleiss' kappa, Krippendorff's alpha, and Python for QC tooling.

Qualifications

• Bachelor's degree in statistics, data science, computer science, or related quantitative field; Master's preferred.
• 6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs.
• IAA methodology expertise including Cohen's kappa, Fleiss' kappa, Krippendorff's alpha.
• QC process design: sampling methodology, escalation workflows, adjudication protocols.
• Python proficiency for QC tooling, metric computation, and statistical analysis.
• Active Secret clearance with TS/SCI eligibility.

Responsibilities

• Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus including metric selection and thresholds.
• Define evaluation framework architecture: test plans, drift targets, and model performance metrics per SOW section 2.9.
• Configure sampling-based quality control across annotation paths during Phase D corpus production.
• Design and implement confidence-threshold escalation routing from automated annotation to senior adjudication.
• Validate quality scoring and IAA computation within the Innodata data layer.
• Support evaluation design for SAM 2 and Frontier model API validation; define what "good enough" looks like quantitatively.
• Produce evaluation framework documentation for the Phase 1 NPP closeout package.

Skills

IAA methodology

QC process design

Python for QC tooling

data quality

active secret clearance

Education

Bachelor's degree in Statistics, Data Science, CS, or related field

Tools

CVAT

Job description

Overview

Innodata (Nasdaq: INOD) is a global data engineering company focused on enabling responsible advancement of AI by providing data, evaluation frameworks, and human expertise to build trusted AI systems at scale. We deliver transferable solutions, platforms, and services for Generative AI builders and adopters, with a 36+ year legacy of high-quality data and outcomes for customers.

About the Program:

Innodata's Federal Practice builds the trusted data layer for critical infrastructure Trust & Safety work. We partner with a leading systems integrator to deliver a modern, governed data services platform in a secure federal (IL4) environment. Over a 20-week phase, you will help stand up a data services storefront, a DataCard governance framework, synthetic data integration, and Databricks write-back capabilities.

About the Role

As the QA/Evaluation Lead, you will own quality and evaluation across the platform. You will design the evaluation framework that measures whether our data services and outputs meet the required standards, build repeatable test and validation processes, and provide the team with an objective read on readiness at each milestone. You will partner with the Delivery Owner and engineering leads to make quality a measurable, demonstrable strength. This role requires rigorous thinking about evaluation and pride in evidence-backed quality.

Key Responsibilities

• Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus — metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds
• Define evaluation framework architecture: test and evaluation plans, IAA targets, drift detection gates, and model performance metrics per SOW Section 2.9
• Configure and operate sampling-based quality control across the self-service and white-glove annotation paths during Phase D corpus production
• Design and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication
• Validate quality scoring and IAA computation within the Innodata data layer
• Support AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation — define what "good enough" looks like quantitatively
• Produce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA

Must-Have Qualifications

• Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.
• 6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs
• IAA methodology expertise — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical
• QC process design: sampling methodology, escalation workflows, adjudication protocols
• Python for QC tooling, metric computation, and statistical analysis
• Active Secret clearance with TS/SCI eligibility

Nice-to-Have Qualifications

• Prior DoD or IC data quality program experience
• CVAT or equivalent annotation platform QC workflow configuration
• Drift detection and model monitoring methodology
• Experience with FMV / video annotation quality standards

The expected hourly salary range for this position is $45 to $50 p/hour, based on experience, skills, and qualifications.