Principal AI Engineer (Context, Agents and Context)

Elastic; Greater London; Hybrid; GBP 120,000 - 190,000; Full time

Benefits offered by this job

Health coverage for you and family

Flexible location and schedule

Generous vacation days

Parental leave (16 weeks) and family 


Volunteer time (40 hours/year)

Charitable giving matching

Job summary

Elastic’s Context Engine team builds the knowledge layer used by AI agents to work with enterprise data in Elasticsearch. We extract knowledge from any data sources into a structured AI Index and expose it to agents through public APIs, MCP tools and framework integrations.

As a Principal AI Engineer, you own the end-to-end production improvement loop: how agents, automations and skills behave, how we observe and evaluate them, and how we ship changes safely.

Qualifications

• Track record of eval-driven product improvement and diagnosing agent/LLM behaviour from traces and feedback.
• Strong backend engineering skills in Python or TypeScript: APIs, stateful workflows, data pipelines.
• Experience running product experiments end to end: instrumentation and interpreting results.
• Experience building agents with state and memory, iterating on prompts, skills and tool behavior in production.
• Experience designing telemetry for AI systems to inform engineering and product decisions.
• Experience with agent frameworks such as LangGraph, CrewAI, Claude Agent SDK or similar.
• Experience with public APIs and evolving data models while maintaining external contracts.
• Practical Elasticsearch experience.

Responsibilities

• Own the production improvement loop for Context Engine: understand extraction automations, retrieval tools and memory behaviour.
• Define iteration on agents and skills safely: versioning, rollout, regression coverage and guardrails.
• Design telemetry to support data-informed engineering decisions: traces, tool calls, knowledge retrieval, dashboards.
• Collaborate with data science to shape evaluation strategies and gate on quality metrics.
• Review designs and PRs, mentor engineers in evaluation-driven development, contribute to the roadmap.

Skills

Backend engineering

Public APIs

Elasticsearch

Agent frameworks

Eval-driven development

Telemetry design

Data science collaboration

Production deployment

Tools

LangGraph

Claude Agent SDK

CrewAI

LangChain

Job description

• The Context Engine team builds the knowledge layer that AI agents use to work with enterprise data in Elasticsearch. We extract knowledge from any data sources into a structured AI Index, serve it to agents through public APIs, MCP tools and framework integrations, and close the loop with agent traces so that what the engine knows improves from real usage. Any agent can use it: Elastic’s own Agent Builder, Claude Code, LangChain and other third-party harnesses
• As a Principal AI Engineer, you own the improvement loop of this product end to end: how agents, automations and skills behave in production, how we observe them, how we evaluate them, and how we ship changes to them safely
• This is a hybrid role at the intersection of engineering, data science, and product. You will write production code, design evaluation and telemetry that product decisions can rest on, and set the technical bar for how the team iterates on agentic behaviour
• You will work alongside data scientists, backend engineers, product, and UX, and your work will show up directly in what customers build on top of Elastic
• The codebase is TypeScript and we build it in the open, so you’ll be shipping code, designs and discussions in public alongside the rest of the Elastic Stack
• Own the production improvement loop for Context Engine: understand how extraction automations, retrieval tools and memory behave, based on offline evaluations and customer conversations and telemetry. You help find the failure modes, fix them, and prove the fix
• Define how we iterate on agents and skills safely: versioning and rollout of prompts, skills and automations, regression coverage, staged and shadow evaluation, and the guardrails that let us change behaviour without breaking customers
• Design the telemetry we need to make data-informed engineering decisions: what to capture from agent traces, tool calls and knowledge retrieval, how it lands in Elasticsearch, and how it feeds evaluation, dashboards and the feedback loop
• Partner with the data science team on evaluation strategy: golden datasets, evaluators to gate on quality, latency and cost
• Raise the bar across the team: review designs and PRs, mentor engineers in eval-driven development, and write the technical proposals that shape the roadmap

Benefits

• Toast to your health: Fully paid health coverage for you and your family, in many locations.
• Craft your calendar: Flexible location and schedule for most roles.
• Create space for you: Distributed by design workforce, plus generous number of vacation days each year.
• Embrace parenthood: Minimum of 16 weeks of parental leave, plus generous family formation benefits.
• Give back your time: 40 hours each year to use toward volunteering with organizations and causes you’re passionate about.
• Amplify your impact: Double your charitable giving — we match donations up to $1500 USD (or local currency equivalent).

Familiarity with MCP, including exposing public MCP servers and toolsA track record of eval-driven product improvement: you have diagnosed agent or LLM behaviour from traces and user feedback, designed the evaluation that exposed the problem, shipped the fix and measured the outcomeA pragmatic, low-ego style suited to a distributed, async-first teamStrong backend engineering skills in either Python or TypeScript: APIs, stateful workflows, data pipelines and production servicesExperience running product experiments end to end: instrumentation, unattended execution, and interpreting resultsDirect experience building agents with state and memory, and iterating on prompts, skills and tool behaviour safely in productionExperience building products with public APIs and evolving data models, and the judgement that comes with maintaining external contracts as a product changesComfort working with data scientists, engineers, and product managers as peers, translating between measurement and shipping, and communicating trade-offs clearlyExperience designing telemetry for AI systems, and using it to make engineering and product decisions10+ years of software engineering experience, with the recent years spent shipping and operating AI-driven products on real production traffic, ideally products with public APIs and data models that had to evolve without breaking customersExperience with agent frameworks such as LangGraph, CrewAI, Claude Agent SDK, or similarAgentic retrieval experience, including knowledge representationPractical Elasticsearch experience