Technical Project Manager

lilt-corporate; United States; Remote; USD 120,000 - 160,000; Full time

Job summary

Lilt-corporate is seeking a program manager for Applied AI initiatives focused on multilingual data collection, human-in-the-loop operations, and LLM evaluation. You will translate AI requirements into executable workflows, coordinate engineering and contributor teams, and drive reliable datasets and evaluation results with measurable quality, speed, and cost.

You will lead data-collection and evaluation efforts from scoping through delivery, define data specs, and align with experts to optimize

Qualifications

• Experience managing complex technical projects involving data operations, AI, machine learning, language technology, or related workflows.
• Strong understanding of LLM training and evaluation concepts (supervised fine-tuning, RLHF, human evaluation, automated evaluation).
• Proficiency in SQL for analyzing delivery velocity, quality metrics, and cost structures.
• Working knowledge of data pipelines, structured data formats, APIs, validation processes, and engineering dependencies.
• Ability to manage complex workflows using Agile, Scrum, or Kanban practices and communicate technical requirements to global teams.
• Experience optimizing delivery economics, such as cost per task or token, while maintaining quality standards.

Responsibilities

• Lead data-collection and evaluation initiatives from technical scoping through implementation, validation, delivery, and retrospective review.
• Convert AI requirements into data specifications, annotation guidance, evaluation plans, acceptance criteria, schemas, metadata requirements, and integration needs.
• Coordinate supervised fine-tuning, preference-data, model-evaluation, safety-testing, and response-ranking workflows with technical specialists and domain experts.
• Partner with engineers on ingestion, annotation tooling, data-generation pipelines, automated checks, system integrations, and issue resolution.
• Operationalize hybrid evaluation approaches that combine human review, automated metrics, and LLM-as-a-judge methods.
• Monitor throughput, quality, completeness, inter-annotator agreement, benchmark discrepancies, delivery risks, and unit economics; lead root-cause analysis when performance falls short.

Skills

Project management

AI/ML experience

SQL

Agile/Scrum/Kanban

Data pipelines

APIs

Cost optimization

Education

Background in CS/DS/Engineering

Tools

Jira

Python

Job description

Role overview

Manage technically complex Applied AI programs involving multilingual data collection, human-in-the-loop operations, and large language model evaluation. The role translates AI requirements into executable workflows, coordinates engineering and contributor teams, and uses operational metrics to deliver reliable datasets and evaluation results at the required quality, speed, and cost.

Responsibilities

• Lead data-collection and evaluation initiatives from technical scoping through implementation, validation, delivery, and retrospective review.
• Convert AI requirements into data specifications, annotation guidance, evaluation plans, acceptance criteria, schemas, metadata requirements, and integration needs.
• Coordinate supervised fine-tuning, preference-data, model-evaluation, safety-testing, and response-ranking workflows with technical specialists and domain experts.
• Partner with engineers on ingestion, annotation tooling, data-generation pipelines, automated checks, system integrations, and issue resolution.
• Operationalize hybrid evaluation approaches that combine human review, automated metrics, and LLM-as-a-judge methods.
• Monitor throughput, quality, completeness, inter-annotator agreement, benchmark discrepancies, delivery risks, and unit economics; lead root-cause analysis when performance falls short.

Requirements

• Experience managing complex technical projects involving data operations, AI, machine learning, language technology, or related workflows.
• Strong understanding of LLM training and evaluation concepts, including supervised fine-tuning, reinforcement learning from human feedback, human evaluation, automated evaluation, LLM-as-a-judge, and red teaming.
• Proficiency in SQL for analyzing delivery velocity, quality metrics, and cost structures.
• Working knowledge of data pipelines, structured data formats, APIs, validation processes, and engineering dependencies.
• Ability to manage complex workflows using Agile, Scrum, or Kanban practices and communicate technical requirements to global teams.
• Experience optimizing delivery economics, such as cost per task or token, while maintaining quality standards.

Nice to have

• Python or scripting experience for data analysis, automation, or evaluation tooling.
• Experience managing specialized subject-matter experts or distributed contributor networks.
• Familiarity with annotation platforms, business-intelligence tools, automated evaluation systems, and Jira.
• Experience delivering multilingual or multimodal data programs, plus fluency in an additional language.
• Background in computer science, data science, engineering, or equivalent practical experience.