AI Evaluation Engineer

Hailey HR; Stockholms kommun; On-site; SEK 600,000 - 900,000; Full time

Job summary

Hailey HR, a Swedish SaaS HR-tech company, is seeking an AI Evaluation Engineer to join our AI development team in Stockholm. You will define how to evaluate AI features, build synthetic data-driven test sets, and develop CI pipelines to ensure reliability and privacy in production.

You will work closely with AI developers, QA engineers, and Product Owners to translate evaluation results into concrete improvements and maintain privacy-friendly monitoring across AI features.

Qualifications

• Master’s degree in CS, DS or engineering physics.
• Around two years of professional experience in software development, data science or test automation with hands-on experience with LLMs.
• Fluency in Swedish and strong English skills, both spoken and written.

Responsibilities

• Define, with the team, what “good” looks like for each AI feature and turn it into measurable criteria.
• Build and maintain test datasets for AI features using synthetic data and realistic edge cases.
• Develop evaluation pipelines that run in CI.
• Set up privacy-friendly monitoring of AI quality in production.
• Turn evaluation results into concrete improvement suggestions for the team.

Skills

Python

AI evaluation

Statistics

Analytical thinking

Swedish (fluency)

Team collaboration

Education

Master’s degree in Computer Science, Data Science, Engineering Physics or equivalent

Tools

CI pipelines

Job description

During the spring of 2020 Hailey HR was founded. We’re a Swedish SaaS HR-tech company that wants to digitalise, simplify and modernise the HR industry with our HR-system - join us on our journey!

About the position

As an AI Evaluation Engineer at Hailey HR, you will be a part of our AI development team - working in close collaboration with AI developers, our QA engineer and Product Owners to make sure the AI features we ship are accurate, reliable and trustworthy.

We are building a range of AI-driven tools for HR, managers and employees. Building AI features is one thing - knowing that they actually work, and keep working when models and prompts change, is another. That is where you come in. You will own how we evaluate our AI, both before a feature is released and once it is live in production.

Our AI features work with sensitive HR data, and protecting our customers' privacy always comes first. That makes evaluating AI quality a real challenge: how do you know an AI feature works well for users without looking at their data? That is the question you will own. If that sounds like a fun problem to solve, keep reading!

What you will do

• Define, together with the team, what “good” looks like for each AI feature and turn it into measurable criteria
• Build and maintain test datasets for our AI features, using synthetic data and realistic edge cases
• Develop evaluation pipelines that run in CI
• Set up privacy-friendly monitoring of AI quality in production
• Turn evaluation results into concrete improvement suggestions for the team

What are we looking for?

• Experienced in working with Python, with an interest in AI evaluation methods and frameworks.
• A person with strong analytical abilities and an interest in statistics and measuring things. Someone curious and eager to learn new technologies and AI models, proactive in taking initiative, and thrives in a collaborative team environment.
• Education: A Master’s degree in Computer Science, Data Science, Engineering Physics or equivalent field.
• Experience: Around two years of professional experience in software development, data science or test automation, with some hands-on experience working with LLMs. Experience in evaluating AI systems is a plus.
• Language: Full professional fluency in Swedish and strong English skills, both spoken and written.

We believe you are

• Flexible and thrives in an innovative and dynamic environment, where solutions are always open to discussion.
• Someone who likes to ask “how do we know this works?” - and then builds the answer.
• Enjoys working in a team, pair programming, and sees the value of code reviews because you work closely with your colleagues in the team.
• Loving table tennis! (or at least find it okay, it’s a part of our culture after all)

More information

• Start: Whenever you’re ready!
• Scope: 100%