AI Evaluation Engineer: Build Quality & Feedback Systems

Cursor; California (MO); On-site; USD 120,000 - 180,000; Full time

Job summary

Cursor is seeking a Software Engineer to join the Agent Quality team, building measurement, evaluation, and feedback-loop infrastructure that improves the core Cursor agent over time.

This role sits at the intersection of product, data, and engineering: instrumenting metrics, shaping quality definitions, and delivering pipelines to analyze agent behavior at scale. You will partner with research, product, and infrastructure to turn insights into improvements.

Qualifications

• Experience building AI evaluation or measurement systems (e.g., evals, experiments, ranking, or quality metrics).
• Strong data acumen with ability to collaborate with data scientists/researchers.
• Up-to-date with model/agent behavior research and industry trends.

Responsibilities

• Designing and building best-in-class AI evaluation system: curated datasets, offline replay, scorers/judges, regression alerts, dashboards.
• Designing feedback loops from real usage: collecting, cleaning, and interpreting user signals to inform model and harness changes.
• Developing analysis tooling and workflows for debugging agent behavior: deep dives on failure modes and clustering themes.
• Improving reliability and guardrails by making quality measurable and actionable: defining good/bad/degraded sessions, alerting, and triage primitives.

Skills

AI evals

Data analysis

Research collaboration

Production engineering

Tools

Python

Experimentation tooling

Job description

Cursor is seeking a Software Engineer to join the Agent Quality team, building measurement, evaluation, and feedback-loop infrastructure that improves the core Cursor agent over time.

This role sits at the intersection of product, data, and engineering: instrumenting metrics, shaping quality definitions, and delivering pipelines to analyze agent behavior at scale. You will partner with research, product, and infrastructure to turn insights into improvements.