AI Test Engineer: Accounting Eval & QA

Sumary ApS

København

On-site

DKK 700,000 - 900,000

Full time

7 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Office in Copenhagen
Real ownership of projects

Job summary

Sumary ApS in Copenhagen is seeking an AI Test Engineer to build and expand evals that verify accounting accuracy on live client books. You will design test cases, scoring, and automated pipelines used on every model update.

You will work with researchers and engineers building the AI CFO, based in our Copenhagen office. The role emphasizes rigorous testing, trustworthy results, and clear communication of findings to ensure high-quality automation across Europe.

Qualifications

  • 5+ years of experience in software engineering, test engineering or ML
  • Built a test harness for LLMs or agents before: datasets, runners, scoring
  • Production-grade test systems with robust, auditable results
  • Rigorous test design to avoid fools and premature conclusions
  • Clear written communication of results and confidence level

Responsibilities

  • Build and grow eval suites across accounting tasks (reconciliation to reporting)
  • Convert real accounting cases into test cases with defined answers
  • Design trustworthy scoring using model-graded checks where appropriate
  • Run evals on every model, prompt and tool change to catch regressions
  • Track agent quality over time and communicate results clearly to the team
  • Help decide when an agent is ready for new tasks

Skills

Test engineering
Software engineering
ML/AI testing
LLM evaluation
CI/CD pipelines

Tools

LLM test harness tooling
Runners & scoring systems

Job description

Sumary ApS in Copenhagen is seeking an AI Test Engineer to build and expand evals that verify accounting accuracy on live client books. You will design test cases, scoring, and automated pipelines used on every model update.

You will work with researchers and engineers building the AI CFO, based in our Copenhagen office. The role emphasizes rigorous testing, trustworthy results, and clear communication of findings to ensure high-quality automation across Europe.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Test Engineer Copenhagen
AI Test Engineer Copenhagen

Sumary ApS • København

On-site
DKK 700,000 - 900,000
Office in Copenhagen
Real ownership of projects
AI-Powered Accounting Implementation Lead
AI-Powered Accounting Implementation Lead

Sumary ApS • København

On-site
DKK 900,000 - 1,100,000
Senior AI Engineer — Build the AI CFO Agent
Senior AI Engineer — Build the AI CFO Agent

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Senior AI Engineer Copenhagen
Senior AI Engineer Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Senior AI-Driven Full-Stack Engineer for Europe Fintech
Senior AI-Driven Full-Stack Engineer for Europe Fintech

Sumary ApS • København

On-site
DKK 900,000 - 1,200,000
Forward Deployed Accountant Copenhagen
Forward Deployed Accountant Copenhagen

Sumary ApS • København

On-site
DKK 900,000 - 1,100,000
Senior Test Automation Engineer – AI-Driven QA & CI/CD
Senior Test Automation Engineer – AI-Driven QA & CI/CD

Twoday A/S • København

On-site
DKK 485,074 - 634,328
Pension
Health insurance
Opportunities for professional development
QA Engineer - Copenhagen, Denmark
QA Engineer - Copenhagen, Denmark

PixelWave Studios • København

On-site
DKK 117,000 - 140,000
Senior QA Engineer - Lead Testing & Automation (Copenhagen)
Senior QA Engineer - Lead Testing & Automation (Copenhagen)

Systematic • København

On-site
DKK 550,000 - 800,000
Employees Club monthly social events
Office catering
Foosball table
+1
QA Tester: Manual & Automated Testing
QA Tester: Manual & Automated Testing

PrimeLedger Associates • Esbjerg

On-site
DKK 122,000 - 146,000
Top locations