AI Test Engineer Copenhagen

Sumary ApS

København

On-site

DKK 700,000 - 900,000

Full time

7 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Office in Copenhagen
Real ownership of projects

Job summary

Sumary ApS in Copenhagen is seeking an AI Test Engineer to build and expand evals that verify accounting accuracy on live client books. You will design test cases, scoring, and automated pipelines used on every model update.

You will work with researchers and engineers building the AI CFO, based in our Copenhagen office. The role emphasizes rigorous testing, trustworthy results, and clear communication of findings to ensure high-quality automation across Europe.

Qualifications

  • 5+ years of experience in software engineering, test engineering or ML
  • Built a test harness for LLMs or agents before: datasets, runners, scoring
  • Production-grade test systems with robust, auditable results
  • Rigorous test design to avoid fools and premature conclusions
  • Clear written communication of results and confidence level

Responsibilities

  • Build and grow eval suites across accounting tasks (reconciliation to reporting)
  • Convert real accounting cases into test cases with defined answers
  • Design trustworthy scoring using model-graded checks where appropriate
  • Run evals on every model, prompt and tool change to catch regressions
  • Track agent quality over time and communicate results clearly to the team
  • Help decide when an agent is ready for new tasks

Skills

Test engineering
Software engineering
ML/AI testing
LLM evaluation
CI/CD pipelines

Tools

LLM test harness tooling
Runners & scoring systems

Job description

Build the evals that prove our agents get accounting right, and catch regressions before our customers do.

About the role

Our agents do real accounting work on live client books, so "it looks right" is not good enough. As an AI Test Engineer, you will build the evals that measure whether they get it right: the test cases, the scoring and the systems that run them on every change.

You will work closely with our research team, with our Applied Accounting Researcher, who decides what correct looks like, and with the engineers building the AI CFO and our agents. You will be based in our Copenhagen office.

What you'll do
  • Build and grow our eval suites across accounting tasks, from reconciliation to reporting and consolidation
  • Turn real cases from accounting firms into test cases with clear, agreed answers
  • Design scoring you can trust, using model-graded checks only where they hold up
  • Run evals on every model, prompt and tool change, and catch regressions before release
  • Track agent quality over time, and make the results easy for the whole team to read
  • Help decide when an agent is good enough to take on a new task
What we're looking for
  • 5 or more years of experience in software engineering, test engineering or machine learning
  • You have built a test harness for LLMs or agents before: the datasets, the runners and the scoring
  • A strong software engineer: your test systems are production code, not a folder of scripts
  • Rigour: you design tests that are hard to fool
  • Clear writing about what results mean, and how sure you are of them
Nice to have
  • Statistics, and experience measuring noisy systems
  • Experience building CI pipelines
  • An interest in accounting or finance
Why Sumary
  • AI is the foundation of everything we build, not a feature bolted on.
  • $25m seed round led by Singular, the largest for accounting software in Europe.
  • Backed by byFounders, Partech and Tenity, and angels from Zendesk, Linear and Qonto.
  • A small team with real ownership. What you build reaches accounting firms and finance teams across Europe.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Test Engineer: Accounting Eval & QA
AI Test Engineer: Accounting Eval & QA

Sumary ApS • København

On-site
DKK 700,000 - 900,000
Office in Copenhagen
Real ownership of projects
Senior AI Engineer Copenhagen
Senior AI Engineer Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Forward Deployed Accountant Copenhagen
Forward Deployed Accountant Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,100,000
Developer Productivity Engineer Copenhagen
Developer Productivity Engineer Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Senior Full-Stack Engineer Copenhagen
Senior Full-Stack Engineer Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Integrations Engineer Copenhagen
Integrations Engineer Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,300,000
Senior AI Engineer — Build the AI CFO Agent
Senior AI Engineer — Build the AI CFO Agent

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Senior Infrastructure Engineer Copenhagen
Senior Infrastructure Engineer Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Seed funding
Small team with ownership
Compliance & Security Engineer Copenhagen
Compliance & Security Engineer Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,300,000
AI-Powered Accounting Implementation Lead
AI-Powered Accounting Implementation Lead

Sumary ApS • København

On-site
DKK 900,000 - 1,100,000
Top locations