Senior AI Quality Architect & Metrics Leader

Lilt; United States; On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Medical benefits

Fringe lifestyle stipend

Paid parental leave

Equity and 401(k) matching

Job summary

LILT in the United States seeks a senior Quality leader to design and scale end-to-end quality frameworks for multiple AI evaluation programs. You will define rubrics, sampling plans, QC gates, and acceptance criteria, partnering with AI, product, and research teams to ensure reproducible results.

You will own tooling and quality instrumentation, supervise annotator operations, and drive data-governance practices to maintain audit-ready deliverables while delivering measurable quality

Qualifications

• Experience designing and running quality frameworks across multiple programs.
• Hands-on with labeling/eval tooling and quality instrumentation.
• 6+ years building/operating quality programs for ML/LLM systems.
• Data governance including privacy/compliance and audit-ready documentation.
• Experience calibrating annotators and drift monitoring in high-throughput evaluation programs.
• Strong grasp of reliability/agreement methods (e.g., Krippendorff’s α, Cohen’s κ).
• Education in CS, Statistics, Data Science, Engineering, Linguistics, or HCI.
• Excellent written and verbal communication; able to translate goals into measurable targets.
• Proficiency in a programming language and modern AI frameworks.
• Fluency in multiple human languages.

Responsibilities

• Design and govern end-to-end quality frameworks across concurrent programs.
• Own tooling, QA infrastructure, and automation for quality measurements.
• Oversee annotator operations, calibration, and drift monitoring to maintain data integrity.
• Lead data governance practices, including privacy constraints and audit-ready documentation.
• Collaborate with AI, product, and research teams to set measurable quality targets and tradeoffs.

Skills

Quality systems leadership

Labeling tooling

Applied AI evaluation

Data governance

Rater operations

Measurement statistics

Programming language proficiency

Multilingual

Education

B.S./M.S. in a quantitative field

Tools

Label Studio

Modern AI frameworks

Job description

LILT in the United States seeks a senior Quality leader to design and scale end-to-end quality frameworks for multiple AI evaluation programs. You will define rubrics, sampling plans, QC gates, and acceptance criteria, partnering with AI, product, and research teams to ensure reproducible results.

You will own tooling and quality instrumentation, supervise annotator operations, and drive data-governance practices to maintain audit-ready deliverables while delivering measurable quality