AI Engineer

Telavox; Stockholms kommun; On-site; SEK 900,000 - 1,300,000; Full time

Job summary

Telavox is seeking an AI Engineer to own evaluation of models used in our transcripts, summaries, and AI agents. You’ll build continuous evaluation pipelines, benchmark models and prompts, and decide which models run in production across our AI products.

You’ll collaborate with backend engineers and the AI domain team to shape priorities with the product manager, ensuring data-driven decisions and measurable quality improvements across regions.

Qualifications

• Hands-on work evaluating LLM/machine-learning model output in a real product.
• Strong Python and data-science tooling, and experience analysing data to make product and business recommendations.
• Experience with prompt design and structured outputs for LLMs in production.
• Ability to communicate findings clearly, in writing and in presentations.
• Proficient in English, both spoken and written.

Responsibilities

• Build and run evaluation pipelines for transcription, summarisation and conversational agents.
• Benchmark candidate models and decide what runs in production with quality and cost as factors.
• Design and evaluate prompts and structured outputs behind our products.

Skills

LLM evaluation

Python

Data science tooling

Prompt design

Communication

English proficiency

Tools

Open models deployment

Self-hosted models

GPU deployment

Job description

Join Telavox as an AI Engineer

Do you want to be the person who decides which models our AI products run on, and prove it with data? Telavox builds AI into the communication platform our customers already use: transcripts and summaries of their phone calls, AI agents that answer calls and messages, and the quality controls around them. We are looking for an AI Engineer who owns how good all of that is, across every AI product we ship.

In this role, placed in Malmö or Stockholm, you'll measure the quality of what our models produce, build the evaluation pipelines that measure it continuously, and choose the models and prompts we run in production. You'll work in a small AI domain with two engineering teams, shaping priorities together with the product manager day to day, and your findings decide what goes live.

If you'd like to take the next step in your career and are excited by the challenge of making AI measurably better, this is your opportunity!

About The Job

Our AI products are live and growing, sold both directly and through telecom operators across Europe. Every one of them depends on models we do not train ourselves, so the choice of model, the prompts, and the quality metrics are decided by evidence from our own calls. This role owns that evidence.

You'll work with the backend engineers who run the production services, with the team building our AI agent platform, and with customers and partners whose requirements shape what good means for them.

Your Day-to-day Work May Include

• Building and running evaluation pipelines for transcription, summarisation and conversational agents: reference sets from real calls, automated metrics, LLM-as-judge, and validation that the metrics themselves can be trusted.
• Benchmarking candidate models, hosted and open, and making the call on what runs in production, with quality as the gate and cost as the constraint.
• Designing and evaluating the prompts and structured outputs behind our products, including the summary instructions customers can define themselves.
• Evaluating models that can be deployed regionally for customers with data-residency requirements.
• Putting quality measures into production, both ours and the ones customers define themselves, and presenting what they show to engineers, product, and leadership.

About You

You're a hands-on engineer who prefers evidence to opinion, and who is comfortable telling the team that the newer model is not better. You build the tooling you need rather than waiting for it; you can read a distribution and spot when a metric is measuring the wrong thing; and you explain your results so that people who did not run the experiment can act on them. You enjoy a small team where your work is visible in the product within weeks.

Key Skills And Experience We're Looking For

• Hands-on work evaluating LLM or machine-learning model output in a real product: building datasets, defining metrics, running benchmarks, and turning them into decisions.
• Strong Python and data-science tooling, and experience analysing data to make product and business recommendations
• Experience with prompt design and structured outputs for LLMs in production
• The ability to communicate findings clearly, in writing and in presentations
• Proficient in English, both spoken and written

Good-to-have Skills

• Speech and audio experience: transcription metrics such as WER and CER, diarization, forced alignment, and timestamp quality
• Java or another backend language, for integrating evaluations into production services
• Experience running open models yourself, such as deploying or evaluating self-hosted speech or language models on GPUs
• Familiarity with conversational AI or real-time voice agents
• Language proficiency in Swedish, Arabic, or other languages spoken in our main markets. Judging the quality of transcripts and agent conversations on customer calls is much easier in a language you speak

Join us at Telavox

Since Telavox's journey started in 2002, we have strengthened our position as the leading forward-thinking communication platform for businesses. As a digital mobile operator, we develop and own our all-in-one communication platform, which integrates telephony, PBX, messaging, meetings, and contact centers, giving businesses a smarter way to connect.

Today, we’re a thriving company with 1.9 billion SEK in revenue and 500+ Telavoxers across nine countries. We embrace AI and automation to push the boundaries of business communication.

We offer flexible work options and adaptable hours, giving you the freedom to balance life and career while staying connected to our vibrant Telavox culture.

Learn more about what we do

Telavox develops world-leading communications platforms with solutions for smart and efficient telephony, PBX, chat and much more. Regardless of whether it involves an entrepreneur-driven startup or a giant among giants, Telavox contributes to making companies more transparent, efficient and accessible.