Senior AI Quality Leader

LILT AI

City of Syracuse (NY)

Hybrid

USD 150,000 - 190,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

LILT is seeking a Quality leader for ML/LLM evaluation and quality infrastructure. You will define KPIs, implement QA gates, and work with engineering and research to deploy instrumentation. The role emphasizes high-volume annotator management, data governance, and collaboration with external partners.

Based in Boston with hybrid work options, you will drive quality across multiple programs and ensure delivery of scalable, auditable QA processes.

Qualifications

  • Education in a quantitative field or equivalent practical experience.
  • 6+ years building or operating quality programs for ML/LLM systems.
  • Experience designing end-to-end quality frameworks across multiple programs.
  • Strong grasp of reliability and agreement methods and statistics.
  • Experience with calibration, adjudication, retraining, drift monitoring in high-throughput evaluation.
  • Hands-on with labeling/eval tooling and deterministic validations.

Responsibilities

  • Build quality workflows ensuring delivered AI outputs meet agreed quality thresholds.
  • Own tooling and quality infrastructure to surface metrics and detect issues automatically.
  • Oversee operational quality at scale across concurrent programs and improve acceptance and reduce rework.

Skills

Multilingual capabilities
Communication with stakeholders
Quality program design
Data governance
Automation tooling

Education

B.S./M.S. in a quantitative field or equivalent practical experience

Tools

Label Studio

Job description

LILT is seeking a Quality leader for ML/LLM evaluation and quality infrastructure. You will define KPIs, implement QA gates, and work with engineering and research to deploy instrumentation. The role emphasizes high-volume annotator management, data governance, and collaboration with external partners.

Based in Boston with hybrid work options, you will drive quality across multiple programs and ensure delivery of scalable, auditable QA processes.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Quality Manager for ML & LLM Pipelines
AI Quality Manager for ML & LLM Pipelines

lilt-corporate • United States

On-site
USD 130,000 - 180,000
Head of Quality & AI Excellence
Head of Quality & AI Excellence

Lilt, Inc. • Boston (MA)

Hybrid
USD 180,000 - 240,000
AI/ML Quality Engineer — LLM Evaluation & QA Automation
AI/ML Quality Engineer — LLM Evaluation & QA Automation

INSPYR Solutions • Cupertino (CA)

Hybrid
USD 91,000 - 105,000
Remote-friendly schedule
Director of AI Quality & Compliance
Director of AI Quality & Compliance

LILT AI • City of Syracuse (NY)

Hybrid
USD 180,000 - 250,000
Remote Director of Quality & AI Compliance
Remote Director of Quality & AI Compliance

LILT • Boston (MA)

Hybrid
USD 180,000 - 240,000
Remote AI QA Engineer for LLMs & Autonomous Agents
Remote AI QA Engineer for LLMs & Autonomous Agents

web • United States

Remote
USD 120,000 - 180,000
Quality Manager remote Analyst
Quality Manager remote Analyst

Executive Staff Recruiters / ESR Healthcare • United States

Remote
USD 90,000 - 130,000
AI-Native QA Engineer for LLM/Agent Quality
AI-Native QA Engineer for LLM/Agent Quality

Newton Research • Massachusetts

On-site
USD 115,000 - 130,000
Salary equity
Remote Lead: Legal QA for AI Training & Compliance
Remote Lead: Legal QA for AI Training & Compliance

SuperAnnotate AI, Inc. • United States

Remote
USD 136,000 - 194,000
Global AI Data Projects Lead
Global AI Data Projects Lead

LILT AI • United States

On-site
USD 120,000 - 180,000