Lead Forward Deployed AI Engineer - Train & Deploy

Revolent Group; United States; Hybrid; USD 180,000 - 260,000; Full time

Job summary

Revolent Group seeks an experienced engineer to build reference GenAI systems and accompanying instructional materials. You will ship working RAG/tools solutions, design rubrics, and craft instructor resources for hands-on learning.

The role emphasizes production-grade implementation, observability, and the ability to explain design choices to clients and students alike. Remote/hybrid work with collaboration across teams is expected.

Qualifications

• 7+ years in software/data/ML engineering.
• Shipped production GenAI systems used by real clients.
• Able to explain technical decisions clearly to varied audiences.
• Teaching, mentoring, or curriculum design experience is a plus.

Responsibilities

• Design and build reference systems that are production-grade and teachable.
• Create build-first curriculum with demonstrable ship-and-evaluate outcomes.
• Develop fair, rubricked assessments aligned to pass/fail standards.
• Collaborate with Programme Lead and Curriculum Designer daily.
• Maintain currency with models and techniques, document decisions.
• Optionally carry material into delivery as founding faculty.

Skills

Python

GenAI systems

Data pipelines

Assessment design

Tools

LangChain/LangGraph

LlamaIndex

OpenAI

Azure OpenAI

Job description

Location: Remote/Hybrid

Reports To: Director, AI Practice

About the Role

This is a build role first. You will produce working reference systems, exercises, assessment rubrics, and instructor materials - not slideware about AI. The strongest candidates have shipped production GenAI systems for real clients and can teach why they made each decision. There is an option, by mutual agreement, to continue into the founding delivery faculty.

What You Will Build

• Reference RAG service (Week 2): a working retrieval-augmented system over a deliberately messy document corpus - hybrid retrieval, re-ranking, structured outputs - built to be instructive, not just correct.
• Reference multi-tool agent (Week 3): a bounded agentic system with tool use, an MCP integration, memory, and human-in-the-loop controls, which students later harden and deploy.
• Evaluation & observability harness (Week 4): golden-dataset tooling, an LLM-as-judge eval suite, regression testing, a tracing dashboard, a guardrails/PII layer, and a prompt-injection red-team exercise.
• Production & LLMOps scaffolding (Week 5): CI/CD for AI systems, cost/latency engineering, secrets handling, monitoring on quality metrics, and the seeded model-deprecation incident with a no-regression migration path.
• Foundations material (Week 1, lighter): model-selection reasoning and the failure-mode taxonomy, including the “break the demo” adversarial exercise.
• Admissions technical assets: the take-home build and its rubric, the solution-design scenarios (including the “don’t build this yet” variant), and the technical deep-dive question and live-coding bank.
• Instructor-facing materials for all of the above: exercise briefs, model solutions, marking rubrics, and teaching notes detailed enough for another senior engineer to deliver.

Key Responsibilities

• Design and build the reference systems above to a production standard, then deliberately instrument them for teaching - surfacing the trade-offs, failure modes, and decision points an FDE must reason about.
• Write applied, build-first curriculum: every module ends in something the learner ships, evaluates, and can defend.
• Design fair, riggable-to-detect assessments and rubrics that hold a genuine standard, in line with the programme’s pass/fail philosophy.
• Work from the existing Curriculum & Delivery Guide and daily lesson outline, flagging load-balance or sequencing issues early (for example, week density) rather than discovering them in delivery.
• Collaborate daily with the Programme Lead and the Curriculum Designer / Technical Writer, handing over clean technical material for instructional polish.
• Keep all content current: select models, frameworks, and techniques that are defensible now, and document choices so they can be versioned as the landscape moves.
• Participate in the end-of-sprint dry run; revise against feedback before any cohort begins.
• Optionally, carry the material into delivery as founding faculty - the people who wrote it teaching it.

Essential Technical Requirements

• Vector stores and embedding models; when not to use RAG
• Agentic systems: tool/function calling, ReAct and plan-and-execute, multi-agent orchestration and its limits
• Context engineering, structured outputs, schema enforcement, prompt design as engineering
• Frameworks such as LangChain/LangGraph, LlamaIndex, or equivalent - with judgement about when to use none

AI Systems, evaluation & LLMOps

• Evaluation: golden datasets, rubric scoring, LLM-as-judge and its biases, regression testing of prompts and pipelines
• Observability and tracing for multi-step agent runs (e.g. LangSmith, Langfuse, Arize, OpenTelemetry-based stacks)
• Guardrails, PII handling, prompt-injection defence, and the agent attack surface
• LLMOps: versioning prompts/models/indexes, CI/CD for AI systems, model routing and cascades
• Cost and latency engineering: caching, batch vs realtime, token economics
• Production monitoring on quality metrics, not just uptime; incident and migration handling

Engineering foundation (assumed across both)

• Expert Python - production-grade: typing, testing, packaging, clean API design (FastAPI or equivalent).
• Cloud & deployment - hands-on with at least one of AWS / Azure / GCP; containers (Docker); IAM, secrets, networking basics; CI/CD pipelines.
• Data - strong SQL; comfort wrangling messy real-world data (CSV, JSON, unstructured text).
• LLM provider APIs - direct experience with Anthropic and/or OpenAI (and ideally Azure OpenAI / Bedrock) in production.
• Security & privacy - practical handling of secrets, data residency, and PII in client or regulated environments.

Experience & Background

• 7+ years in software / data / ML engineering, with at least 2 years building GenAI or LLM-based systems.
• Has shipped at least one production GenAI system that real users or clients depended on - not only prototypes or notebooks.
• Has built both the application layer (RAG/agents) and the surrounding systems layer (evals, deployment, monitoring) - the combined profile this role requires.
• Can explain a technical decision clearly to a mixed audience and write to a standard suitable for client-facing and instructional material.
• Highly desirable: financial-services or other regulated-industry exposure (aligned to our client base); prior teaching, mentoring, bootcamp, or curriculum-design experience; forward-deployed or client-embedded delivery experience.