Senior AI Engineer, Agents — Bot Quality & Evaluation (Python)

Bliro GmbH; United States; Hybrid; USD 123,000 - 168,000; Full time

Benefits offered by this job

Meaningful equity

Hybrid setup in Munich

Job summary

Bliro GmbH builds an AI assistant that handles desk work for field sales teams, with over one million customer touchpoints documented. The role focuses on owning agent quality from prompting and context to model evaluation and benchmarking.

You'll ship initial improvements, create eval sets, and set a repeatable process to drive measurable gains while working in a small, ownership‑driven team with a hybrid setup in Munich.

Qualifications

• Evidence of shipping LLM agents to real users with measured improvements.
• Strong Python skills and ability to debug agent behavior across prompts, tools, context, and model choice.
• Build eval pipelines with LLM judges and use statistics to distinguish real gains from noise.

Responsibilities

• In 30 days, map agents end to end, review real user conversations, and ship an initial improvement.
• In 60 days, build eval sets and metrics, benchmark models on quality, latency, and cost, and ship a measurable improvement.
• In 90 days, establish a repeatable improvement model and set the roadmap for agent quality.

Skills

Python

LLM evaluation

Debug agent behavior

Job description

Bliro GmbH builds an AI assistant that handles desk work for field sales teams, with over one million customer touchpoints documented. The role focuses on owning agent quality from prompting and context to model evaluation and benchmarking.

You'll ship initial improvements, create eval sets, and set a repeatable process to drive measurable gains while working in a small, ownership‑driven team with a hybrid setup in Munich.