AI Agent Quality Scientist — Drive Evaluation & Impact

Lovable; Stockholms kommun; On-site; SEK 750,000 - 1,100,000; Full time

Job summary

Lovable, a Stockholm-based company, seeks a Data Scientist to own AI agent quality metrics and build eval systems. You will design experiments, trace agent behavior, and drive measurable improvements in success, completion, and error rates.

You’ll collaborate closely with the agent engineering team to translate telemetry into fixes and continuously enhance agent performance. The role emphasizes strong SQL and Python skills, applied statistics, and the ability to design noisy A/B tests.

Qualifications

• Proficient in SQL and Python with strong statistics background.
• Experience with designing and running A/B tests for agent changes.
• Ability to build eval systems and observe model behavior.
• Comfortable turning telemetry into concrete fixes for the agent.

Responsibilities

• Define and own metrics for agent quality (success, completion, error rates).
• Build eval systems and experiment framework (A/B rollout) to catch regressions.
• Turn traces and telemetry into concrete fixes with the agent engineering team.
• Develop tooling to evaluate the evolving agent continuously.
• Set the bar for agent behavior where there is no fixed key.

Skills

SQL

Python

Applied statistics

Experimentation

A/B testing

Data analysis

Analytical mindset

Curiosity

Tools

BigQuery

PubSub

OTEL tracing

Braintrust

Hex

Lovable Apps

Job description

Lovable, a Stockholm-based company, seeks a Data Scientist to own AI agent quality metrics and build eval systems. You will design experiments, trace agent behavior, and drive measurable improvements in success, completion, and error rates.

You’ll collaborate closely with the agent engineering team to translate telemetry into fixes and continuously enhance agent performance. The role emphasizes strong SQL and Python skills, applied statistics, and the ability to design noisy A/B tests.