QA Engineer: AI Evaluation in Legal Tech

Newcode; Oslo; Hybrid; NOK 650,000 - 900,000; Full time

Job summary

Newcode.ai is a fast-growing legal tech company applying agentic AI to transform how legal work is done. As a QA Engineer, you will own the evaluation frameworks and metrics that define AI product quality in real-world workflows.

You will collaborate with Software Engineering and Legal Engineering to translate legal use cases into rigorous evaluation criteria, build automation, and ensure high standards of reliability across our AI systems.

Qualifications

• Strong software engineering or technical background, with experience working on AI/ML systems.
• Experience building testing, evaluation, or quality frameworks for AI-powered products.
• Strong Python and/or similar programming experience.
• Understanding of LLMs, generative AI, retrieval systems, agents, and/or AI evaluation methodologies.

Responsibilities

• Design and build automated and human-in-the-loop evaluation frameworks for AI-powered products and workflows.
• Develop metrics and benchmarks to measure accuracy, reliability, consistency, and overall AI quality.
• Build evaluation datasets, test cases, and regression suites for complex legal use cases.
• Analyze model outputs to identify failure modes, patterns, and opportunities for improvement.
• Work closely with Legal Engineers to translate real-world legal workflows into rigorous evaluation criteria.
• Partner with Software Engineers to integrate evaluations directly into product and development workflows.
• Establish automated testing and monitoring to identify quality regressions before they reach customers.
• Evaluate changes to models, prompts, retrieval systems, agents, and workflows.
• Develop tooling that makes AI quality measurable and visible across the organization.
• Investigate unexpected model behavior and determine root causes.
• Help establish standards for what "good" looks like across different legal use cases.

Skills

Python

AI/ML systems

Testing frameworks

Evaluation frameworks

Software engineering

Cross-functional collaboration

Communication skills

Problem solving

Tools

Retrieval systems

NLP pipelines

Job description

Newcode.ai is a fast-growing legal tech company applying agentic AI to transform how legal work is done. As a QA Engineer, you will own the evaluation frameworks and metrics that define AI product quality in real-world workflows.

You will collaborate with Software Engineering and Legal Engineering to translate legal use cases into rigorous evaluation criteria, build automation, and ensure high standards of reliability across our AI systems.