Lead LLM Engineer: Workflow Architect and Evaluator

Fully Powered

Los Angeles (CA)

On-site

USD 120,000 - 190,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Fully Powered AI, based in Los Angeles, seeks a leader to drive experiments and engineering for dependable AI-generated workflows. You will shape how workflows are extracted from interviews and docs, build robust evaluations, and compare prompting, retrieval, and fine-tuning against baselines.

You’ll design datasets with clear provenance and help translate results into maintainable features. We welcome candidates who can trace failures to data, context, or code, and who can run model training

Qualifications

  • You have built LLM applications and can trace an output failure back to the data, context, instructions, model or surrounding code.
  • You have run model training or fine-tuning experiments and can explain the baseline, data choices and measured result.
  • You write strong Python and can build reproducible evaluation and data pipelines.
  • You can distinguish a persuasive demonstration from a repeatable improvement, and make that distinction clear to the team.

Responsibilities

  • Lead the experiments, evaluations and engineering that make generated workflows more useful and dependable.
  • Improve how we extract workflow structure from interviews, documents and examples, including missing information and conflicting instructions.
  • Build evaluations for generated workflows: completeness, factual grounding, exception handling and whether the work meets the agreed standard.
  • Compare prompting, retrieval, structured generation and fine-tuning against explicit baselines, including quality, latency and cost.
  • Design datasets and experiments for specialized models, with clear provenance and separation of training and evaluation material.
  • Turn promising results into maintainable product features, document failure modes and work with implementation teams to test improvements.

Skills

Python
LLM applications
Evaluation pipelines
Experiment design

Tools

Experiment tracking tools

Job description

Fully Powered AI, based in Los Angeles, seeks a leader to drive experiments and engineering for dependable AI-generated workflows. You will shape how workflows are extracted from interviews and docs, build robust evaluations, and compare prompting, retrieval, and fine-tuning against baselines.

You’ll design datasets with clear provenance and help translate results into maintainable features. We welcome candidates who can trace failures to data, context, or code, and who can run model training

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead LLM Engineer
Lead LLM Engineer

Fully Powered • Los Angeles (CA)

On-site
USD 120,000 - 190,000
Lead/Staff Engineer - Applied AI
Lead/Staff Engineer - Applied AI

US based SaaS company that offers AI-powered business operating system • United States

Remote
USD 180,000 - 280,000
Lead Architect, AI Workflows & Systems
Lead Architect, AI Workflows & Systems

Fully Powered • Los Angeles (CA)

On-site
USD 140,000 - 210,000
AI Engineer — LLM Production & Evaluation
AI Engineer — LLM Production & Evaluation

LawPro.ai • Virginia

On-site
USD 150,000 - 190,000
Senior AI Engineer — LLM Evaluation & Production Systems
Senior AI Engineer — LLM Evaluation & Production Systems

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
Production AI Engineer: LLM Workflows & Guardrails
Production AI Engineer: LLM Workflows & Guardrails

DataDrone Inc. • United States

Remote
USD 140,000 - 230,000
AI Engineer - NC
AI Engineer - NC

LawPro.ai • North Carolina

On-site
USD 140,000 - 190,000
AI Engineer - VA
AI Engineer - VA

LawPro.ai • Virginia (MN)

On-site
USD 140,000 - 200,000
Client-Facing AI Workflow Engineer
Client-Facing AI Workflow Engineer

Fully Powered • Los Angeles (CA)

On-site
USD 120,000 - 180,000
Senior AI Engineer — RL & LLM Evaluation Lead
Senior AI Engineer — RL & LLM Evaluation Lead

EnableComp LLC • United States

Remote
USD 120,000 - 180,000