AI Engineer, Quality & Evals (m/f/x)

Meyandy LLC; Berlin; Vor Ort; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

Cortea AI in Berlin is seeking an AI Engineer focused on Quality & Evaluations to build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows.

You will design data architectures, write production code, and improve the quality, cost, reliability, and performance of AI agents across multimodal workflows. Join us in shaping AI-driven audits from Berlin.

Qualifikationen

• Strong Python and backend engineering experience.
• Solid understanding of how LLMs and agent systems are evaluated.
• Experience deploying and operating systems in the cloud, ideally on GCP.
• Hands-on experience building end-to-end retrieval/ML pipeline evaluation systems and using LLM observability tools (Braintrust, MLflow, Langfuse, Weights & Biases).
• Comfort with analytical databases, data warehouses, columnar stores, and high-volume event/trace data.
• Knowledge of system design, reliability, observability, monitoring, logging, debugging, and trade-offs.

Aufgaben

• Build and operate technical systems around AI agents focusing on data infrastructure, evaluation, observability, and optimization.
• Develop online and offline evaluation systems for LLM agents using golden datasets, ground-truth data, and human review workflows.
• Create automated quality gates to test changes before production.
• Analyze large volumes of agent traces in databases to identify failure modes, regressions, latency and cost opportunities.
• Build data retention and replay mechanisms for long-term analysis of production agent behavior.
• Manage observability tools for tracing, monitoring, debugging and experiment management.
• Collaborate with backend engineers to improve speed and reliability of retrieval and reasoning agents.

Kenntnisse

Python

Backend engineering

LLM evaluation

Data pipelines

Observability

Cloud (GCP)

Tools

Braintrust

MLflow

Langfuse

Weights & Biases

Jobbeschreibung

About us We’re Cortea, a Berlin startup transforming audits with AI. Manual, document-heavy audits waste expert time while demand keeps rising. Our AI-powered software and specialized AI agents remove the repetitive work so auditors can focus on judgment. Backed by top-tier VCs with >15m EUR funding, with a working product and paying customers, we’re rapidly scaling. We value first-principles thinking, speed, trust, and kindness. We build side by side in our Berlin office.

Your Role

We are looking for an engineer with strong backend, data, and AI systems experience to build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows. This role sits at the intersection of backend engineering, data infrastructure, and AI quality. You will build the evaluation systems that power our multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines. You’ll work at the edge of applied AI and information retrieval, building multimodal agentic pipelines and solving hard context and agent-harness engineering problems. This is not a traditional analytics, BI, or dashboarding role. You should expect to write production code, design data architecture, work inside backend systems, and directly improve the quality, cost, reliability, and performance of LLM-based agents.

What you’ll do

You will help build and operate the technical systems around our AI agents, with a focus on data infrastructure, evaluation, observability, and optimization.

• Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results.
• Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.
• Analyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities.
• Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior.
• Manage observability tools for tracing, monitoring, debugging, and experiment management of our audit agents.
• Team up with backend engineers to improve the speed and reliability of our retrieval and reasoning agents.

You will fit into the role if you...

• Have strong Python and/or backend engineering experience.
• Have a solid understanding of how LLM and agent systems are evaluated—including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics—and can reason about when each approach is appropriate.
• Have deployed and operated systems in the cloud, ideally on GCP.
• Have hands‑on experience building end-to-end retrieval or ML pipeline evaluation systems and using LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.
• Are comfortable working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data.
• Understand system design, reliability, observability, monitoring, logging, debugging, and operational trade-offs.
• Bring senior-level engineering judgment: you can make architectural decisions, communicate trade-offs, and build systems that other engineers can extend.
• Are comfortable with ambiguity, able to reason from first principles, and excited to build infrastructure for AI systems that are actively used in production.

Nice‑to‑haves that are a plus

• Designing data pipelines, ETL/ELT workflows, event‑processing systems, or feedback loops for production data.
• Building infrastructure around LLM‑based products or agentic systems, including optimizing LLM usage, context windows, reasoning tokens, or model selection.
• Working with production traces from complex distributed systems.
• Building internal platforms for engineers, domain exper…

AI Engineer, Quality & Evals (m/f/x) — Cortea AI, Berlin.

Mots-clés : Engineering.