Senior Applied AI Quality Leader

lilt-corporate; Irvine (CA); Hybrid; USD 140,000 - 200,000; Full time

Job summary

LILT is seeking a Quality Lead for Applied AI evaluation to design and manage end-to-end quality programs across multiple ML/LLM initiatives. You will build workflows, tooling, and metrics to ensure high-quality translations and AI outputs.

The role requires strong statistical and operational expertise, with collaboration across AI, product, and external partners. Based in Boston with hybrid work options, this position emphasizes governance, transparency, and scalable quality systems to improve

Qualifications

• Education: B.S./M.S. in a quantitative field (CS, Statistics, Data Science, Engineering, Linguistics, HCI) or equivalent practical experience.
• Applied AI / evaluation experience: 6+ years building or operating quality programs for ML/LLM systems (data labeling, benchmark creation, model evaluation, or human-in-the-loop pipelines).
• Quality systems leadership: Design and run end-to-end quality frameworks across multiple concurrent programs.
• Measurement & statistics: Reliability and agreement methods, power/sample-size intuition, error analysis, executive reporting.
• Rater/annotator operations: Calibration, adjudication, drift monitoring, integrity/fraud detection in high-throughput evaluation programs.

Responsibilities

• Ensure Workflows Are Built for Quality with measurable outcomes and audit-ready records.
• Own Technical Tooling & Quality Infrastructure to track defect rates, acceptance, rework, and throughput.
• Own Operational Quality at Scale to ensure programs meet quality targets within cost and time.
• Delivery sign-off: ensure go/no-go decisions and readiness memos before shipping, with post-sign-off quality control.
• Own Customer Quality Remediation: close disputes quickly with quantitative analysis and tracked improvements.

Skills

Applied AI evaluation

Quality systems leadership

Measurement & statistics

Rater/annotator operations

Tooling & automation

Data governance

Stakeholder management

Multilingual fluency

Programming knowledge

Education

B.S./M.S. in a quantitative field

Tools

Label Studio

Job description

LILT is seeking a Quality Lead for Applied AI evaluation to design and manage end-to-end quality programs across multiple ML/LLM initiatives. You will build workflows, tooling, and metrics to ensure high-quality translations and AI outputs.

The role requires strong statistical and operational expertise, with collaboration across AI, product, and external partners. Based in Boston with hybrid work options, this position emphasizes governance, transparency, and scalable quality systems to improve