AI Agent Quality Scientist

Lovable; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

Lovable is seeking a data scientist to own how we measure and improve our AI agent. You’ll define metrics for success, completion, and error rates, and build eval systems and experiments to assess agent changes.

You’ll turn agent traces and telemetry into concrete fixes, collaborating closely with the agent engineering team. You’ll also build automated evaluation tooling for continuous improvement of agent behavior, using strong SQL, Python, and statistics to drive robust results.

Qualifications

• Must have strong SQL and Python, applied statistics, and experimentation.
• Experience or strong interest in LLM evaluation and observability: building evals, scoring outputs, tracing agent behavior, and catching regressions.
• Able to design A/B tests for agent changes where outcomes are noisy.
• Build systems and agents that produce insights continuously, not one-off analyses.

Responsibilities

• Define and own the metrics for agent quality: success, completion, error rates, and related behaviors.
• Build eval systems and experiment framework to decide if an agent change ships (A/B rollout that catches regressions).
• Turn agent traces and telemetry into concrete fixes with the agent engineering team.
• Build tooling and agents that produce evaluations continuously as the agent evolves.
• Set the bar for judging agent behavior where there is no exact answer key.

Skills

SQL

Python

Applied statistics

Experimentation

LLM evaluation & observability

A/B testing

Tools

Braintrust

OTEL tracing

BigQuery

PubSub

Hex

Lovable Apps

Job description

Lovable is seeking a data scientist to own how we measure and improve our AI agent. You’ll define metrics for success, completion, and error rates, and build eval systems and experiments to assess agent changes.

You’ll turn agent traces and telemetry into concrete fixes, collaborating closely with the agent engineering team. You’ll also build automated evaluation tooling for continuous improvement of agent behavior, using strong SQL, Python, and statistics to drive robust results.