Principal AI Engineer, Context Engine & Agent Telemetry

Elastic; Polska; Hybrid; PLN 369,000 - 584,000; Full time

Benefits offered by this job

Health coverage

Flexible schedules

Generous vacation days

Donation matching

Volunteer hours

Job summary

The Context Engine team builds the knowledge layer that AI agents use to work with enterprise data in Elasticsearch. As a Principal AI Engineer, you own the improvement loop end to end: how agents, automations and skills behave in production, how we observe them, how we evaluate them, and how we ship changes to them safely.

This is a hybrid role at the intersection of engineering, data science, and product, and you will write production code, design evaluation and telemetry that product

Qualifications

• 10+ years of software engineering experience, with the recent years spent shipping and operating AI‑driven products on real production traffic, ideally products with public APIs and data models that had to evolve without breaking customers.
• A track record of eval‑driven product improvement: you have diagnosed agent or LLM behaviour from traces and user feedback, designed the evaluation that exposed the problem, shipped the fix and measured the outcome.
• Direct experience building agents with state and memory, and iterating on prompts, skills and tool behaviour safely in production.
• Familiarity with MCP, including exposing public MCP servers and tools.
• Experience designing telemetry for AI systems, and using it to make engineering and product decisions.
• Experience running product experiments end to end: instrumentation, unattended execution, and interpreting results.
• Experience building products with public APIs and evolving data models, and the judgement that comes with maintaining external contracts as a product changes.
• Strong backend engineering skills in either Python or TypeScript: APIs, stateful workflows, data pipelines and production services.
• Comfort working with data scientists, engineers, and product managers as peers, translating between measurement and shipping, and communicating trade‑offs clearly.
• A pragmatic, low‑ego style suited to a distributed, async‑first team.

Responsibilities

• Own the production improvement loop for Context Engine: understand how extraction automations, retrieval tools and memory behave, based on offline evaluations and customer conversations and telemetry. You help find the failure modes, fix them, and prove the fix.
• Define how we iterate on agents and skills safely: versioning and rollout of prompts, skills and automations, regression coverage, staged and shadow evaluation, and the guardrails that let us change behaviour without breaking customers.
• Design the telemetry we need to make data‑informed engineering decisions: what to capture from agent traces, tool calls and knowledge retrieval, how it lands in Elasticsearch, and how it feeds evaluation, dashboards and the feedback loop.
• Partner with the data science team on evaluation strategy: golden datasets, evaluators to gate on quality, latency and cost.
• Raise the bar across the team: review designs and PRs, mentor engineers in eval‑driven development, and write the technical proposals that shape the roadmap.

Skills

Software engineering

AI systems production

Backend engineering

Python

TypeScript

Data science collaboration

Telemetry design

Product experimentation

Public APIs

Agent frameworks

Tools

MCP

Elasticsearch

LangChain

LangGraph

CrewAI

Claude Agent SDK

Job description

The Context Engine team builds the knowledge layer that AI agents use to work with enterprise data in Elasticsearch. As a Principal AI Engineer, you own the improvement loop end to end: how agents, automations and skills behave in production, how we observe them, how we evaluate them, and how we ship changes to them safely.

This is a hybrid role at the intersection of engineering, data science, and product, and you will write production code, design evaluation and telemetry that product