Senior AI Engineer: Agent Quality & Evaluation

Bliro; New Munich (MN); Hybrid; USD 101,000 - 146,000; Full time

Benefits offered by this job

Equity stake

30 days PTO

Hybrid work (Munich/remot e)

Free meals and drinks

Deutschlandticket

Quality equipment

Wellpass

Job summary

Bliro is seeking a senior AI engineer to own agent quality for our chat and voice assistants used by field sales teams. You will map the agent ecosystem, define metrics, and drive measurable improvements in performance, latency, and cost. Join a small team delivering real user value and shaping the future of our AI platform.

You will focus on prompting, context, benchmarking, and evaluating models, with an emphasis on practical, data-driven results and a strong technical background.

Qualifications

• Ownership of shipping LLM agents to real users with data proving improvements.
• Pursued a technical major at top technical universities (TUM, ETH, KIT, RWTH, EPFL).
• Strong Python skills and ability to debug agent behavior across prompts, tools, context and model choice.
• Experience building eval pipelines, LLM judges and model benchmarks with statistics to prove gains.
• Ability to read raw conversations and turn findings into test cases, fixes and clear priorities.

Responsibilities

• Map the chat and voice agents end to end within 30 days.
• Own agent quality measurement and build eval sets and metrics.
• Establish a repeatable model for improving agent quality and run evals on prompts and harness changes.

Skills

Python

Debugging agent behavior

Eval pipelines

LLM benchmarks

Agent frameworks

Multilingual agents

Realtime voice agents

AI coding tools

Education

Technical university degree

Job description

Bliro is seeking a senior AI engineer to own agent quality for our chat and voice assistants used by field sales teams. You will map the agent ecosystem, define metrics, and drive measurable improvements in performance, latency, and cost. Join a small team delivering real user value and shaping the future of our AI platform.

You will focus on prompting, context, benchmarking, and evaluating models, with an emphasis on practical, data-driven results and a strong technical background.