Lead LLM Engineer

Licorne Society; Paris; Sur place; EUR 90 000 - 130 000; Plein temps

Résumé du poste

Licorne Society is partnering with a growing AI startup to hire a Lead LLM Engineer who can make AI outputs reliable, fast, and indispensable in real workflows. You will own the architecture and ensure quality across key use cases.

You will design production-grade pipelines, build evaluation systems, and drive fast iterations from real data. Collaboration with product and founders will shape what to build and why.

Qualifications

• You’ve built systems used in production (not demos).
• You understand RAG, tools, agents, structured outputs.
• You can design full pipelines, not just prompts.
• You know how to define quality metrics and create datasets from real usage.
• You run continuous evals to prevent regressions.
• You can trace issues across retrieval, prompts, and model behavior.
• You move quickly from problem to solution using logs and data.

Responsabilités

• Design and evolve LLM/agent architecture.
• Own output quality across key use cases.
• Build evaluation systems (datasets, metrics).
• Drive fast iteration from production data.
• Improve retrieval, reasoning, and tool usage.
• Ensure production reliability (latency, failure modes, fallback).
• Collaborate with product and founders on what to build and why.

Connaissances

Production systems

RAG & tools

Pipelines design

Quality metrics

Dataset creation

Continuous evals

Debugging failures

Tracing & logs

Fast iteration

Decision making

Outils

Python

FastAPI

PostgreSQL

Google Cloud

LangGraph

LangChain

PostHog

Langfuse

Azure OpenAI

Description du poste

Licorne Society a été missionné par une startup IA en pleine croissance pour les aider à trouver leur Lead LLM Engineer.

What you will own

You will be responsible for one thing:
Make our AI outputs reliable, fast, and indispensable in real workflows.
Concretely:

• Design and evolve our LLM / agent architecture
• Own output quality across key use cases (emails, document analysis, etc.)
• Build evaluation systems (datasets, metrics, regression detection)
• Drive fast iteration loops from production data
• Improve retrieval, reasoning, and tool usage
• Ensure production reliability (latency, failure modes, fallback)
• Work directly with product + founders on what to build and why

What this role is really about

Most teams fail because:

• they don’t know what “good output” means
• they don’t have evals
• they iterate randomly
• they overuse agents

Your job is to fix that.
You will turn:

• vague user problems
• into structured AI systems
• with measurable performance
• that improve every week

What you need to be excellent at

1. Shipping real LLM systems

• You’ve built systems used in production (not demos)
• You understand RAG, tools, agents, structured outputs
• You can design full pipelines, not just prompts

2. Evaluation-driven development

• You know how to define quality metrics
• You build datasets from real usage
• You run continuous evals to prevent regressions

3. Debugging complex failures

• You can trace issues across:
• retrieval
• prompts
• model behavior
• You don’t guess — you isolate and fix

4. Speed of iteration

• You move from problem improvement in hours or days, not weeks
• You use logs, traces, and data — not intuition alone

5. Strong judgment

• You know when to:
• use an agent vs a pipeline
• add complexity vs simplify
• You optimize for reliability and user value, not novelty

What we don’t care about

• Number of years of experience
• Whether you’ve used a specific framework
• Fancy research credentials

If you can build, debug, and improve real systems, you’re a fit.

What success looks like (first 90 days)

• Clear eval framework for core use cases
• Measurable improvement in output quality
• Faster iteration cycles across the team
• Reduced hallucinations / failures
• Stronger system architecture decisions

Stack (context, not requirements)

• Python (FastAPI)
• Postgres
• Google Cloud
• LangGraph / LangChain (evolving)
• PostHog (product analytics)
• Langfuse (LLM traces)
• LLM APIs (Azure OpenAI)