Applied AI Program Manager: Data Operations & Evaluation

lilt-corporate; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

Lilt-corporate is seeking a program manager for Applied AI initiatives focused on multilingual data collection, human-in-the-loop operations, and LLM evaluation. You will translate AI requirements into executable workflows, coordinate engineering and contributor teams, and drive reliable datasets and evaluation results with measurable quality, speed, and cost.

You will lead data-collection and evaluation efforts from scoping through delivery, define data specs, and align with experts to optimize

Qualifications

• Experience managing complex technical projects involving data operations, AI, machine learning, language technology, or related workflows.
• Strong understanding of LLM training and evaluation concepts (supervised fine-tuning, RLHF, human evaluation, automated evaluation).
• Proficiency in SQL for analyzing delivery velocity, quality metrics, and cost structures.
• Working knowledge of data pipelines, structured data formats, APIs, validation processes, and engineering dependencies.
• Ability to manage complex workflows using Agile, Scrum, or Kanban practices and communicate technical requirements to global teams.
• Experience optimizing delivery economics, such as cost per task or token, while maintaining quality standards.

Responsibilities

• Lead data-collection and evaluation initiatives from technical scoping through implementation, validation, delivery, and retrospective review.
• Convert AI requirements into data specifications, annotation guidance, evaluation plans, acceptance criteria, schemas, metadata requirements, and integration needs.
• Coordinate supervised fine-tuning, preference-data, model-evaluation, safety-testing, and response-ranking workflows with technical specialists and domain experts.
• Partner with engineers on ingestion, annotation tooling, data-generation pipelines, automated checks, system integrations, and issue resolution.
• Operationalize hybrid evaluation approaches that combine human review, automated metrics, and LLM-as-a-judge methods.
• Monitor throughput, quality, completeness, inter-annotator agreement, benchmark discrepancies, delivery risks, and unit economics; lead root-cause analysis when performance falls short.

Skills

Project management

AI/ML experience

SQL

Agile/Scrum/Kanban

Data pipelines

APIs

Cost optimization

Education

Background in CS/DS/Engineering

Tools

Jira

Python

Job description

Lilt-corporate is seeking a program manager for Applied AI initiatives focused on multilingual data collection, human-in-the-loop operations, and LLM evaluation. You will translate AI requirements into executable workflows, coordinate engineering and contributor teams, and drive reliable datasets and evaluation results with measurable quality, speed, and cost.

You will lead data-collection and evaluation efforts from scoping through delivery, define data specs, and align with experts to optimize