AI Evaluation Engineer: Shape Live Models & Metrics

Telavox; Malmö kommun; On-site; SEK 780,000 - 1,100,000; Full time

Job summary

Telavox is seeking an AI Engineer to define which models power our AI features, measure quality, and build evaluation pipelines across transcripts, summaries, and conversational agents. You will work with backend engineers, AI agent platform teams, and customers to shape production decisions in a data-driven way.

The role focuses on Python-based evaluation, model benchmarking, and prompt design to deliver measurable improvements in product quality across Europe, with Malmo/Stockholm placements

Qualifications

• Hands-on work evaluating LLM or machine-learning model output in a real product.
• Strong Python and data-science tooling, and experience analysing data to make product and business recommendations
• Experience with prompt design and structured outputs for LLMs in production
• The ability to communicate findings clearly, in writing and in presentations
• Proficient in English, both spoken and written

Responsibilities

• Building and running evaluation pipelines for transcription, summarisation and conversational agents: reference sets from real calls, automated metrics, LLM-as-judge, and validation that the metrics themselves can be trusted.
• Benchmarking candidate models, hosted and open, and making the call on what runs in production, with quality as the gate and cost as the constraint.
• Designing and evaluating the prompts and structured outputs behind our products, including the summary instructions customers can define themselves.
• Evaluating models that can be deployed regionally for customers with data-residency requirements.
• Putting quality measures into production, both ours and the ones customers define themselves, and presenting what they show to engineers, product, and leadership.

Skills

LLM evaluation

Python

Prompt design

Communication

English

Tools

Java

Job description

Telavox is seeking an AI Engineer to define which models power our AI features, measure quality, and build evaluation pipelines across transcripts, summaries, and conversational agents. You will work with backend engineers, AI agent platform teams, and customers to shape production decisions in a data-driven way.

The role focuses on Python-based evaluation, model benchmarking, and prompt design to deliver measurable improvements in product quality across Europe, with Malmo/Stockholm placements