Senior AI Engineer, Agents (Python) (Fully Remote)

Bliro GmbH; United States; Hybrid; USD 123,000 - 168,000; Full time

Benefits offered by this job

Meaningful equity

Hybrid setup in Munich

Job summary

Bliro GmbH builds an AI assistant that handles desk work for field sales teams, with over one million customer touchpoints documented. The role focuses on owning agent quality from prompting and context to model evaluation and benchmarking.

You'll ship initial improvements, create eval sets, and set a repeatable process to drive measurable gains while working in a small, ownership‑driven team with a hybrid setup in Munich.

Qualifications

• Evidence of shipping LLM agents to real users with measured improvements.
• Strong Python skills and ability to debug agent behavior across prompts, tools, context, and model choice.
• Build eval pipelines with LLM judges and use statistics to distinguish real gains from noise.

Responsibilities

• In 30 days, map agents end to end, review real user conversations, and ship an initial improvement.
• In 60 days, build eval sets and metrics, benchmark models on quality, latency, and cost, and ship a measurable improvement.
• In 90 days, establish a repeatable improvement model and set the roadmap for agent quality.

Skills

Python

LLM evaluation

Debug agent behavior

Job description

• Own agent quality for Bliro's chat and voice agents, from prompting and context to model benchmarking and evals.
• Coordinate and evaluate models rather than training them, working within a small team of specialists.

What You'll Do:

• In 30 days, map agents end to end, review real user conversations, and ship an initial improvement.
• In 60 days, build eval sets and metrics, benchmark models on quality, latency, and cost, and ship a measurable improvement.
• In 90 days, establish a repeatable improvement model and set the roadmap for agent quality.

Who You Are:

• You have evidence of shipping LLM agents to real users and proving they got better with data.
• You have strong Python and can debug agent behavior across prompts, tools, context, and model choice.
• You build eval pipelines with LLM judges and use statistics to tell real gains from noise.

Why Bliro:

• We build the human interface for millions of field sellers with a culture of ownership and urgency.
• We offer a strong base salary plus meaningful equity and a hybrid setup in Munich.

Bliro builds an AI assistant that handles desk work for field sales teams, with over one million customer touchpoints documented. Backed by leading investors and trusted by German Mittelstand companies, it's a small, high-performing team that values ownership and continuous improvement.