Lead AI Agent Optimization & Evaluation Engineer

Reddit, Inc.; San Francisco (CA); On-site; USD 180,000 - 300,000; Full time

Benefits offered by this job

Comprehensive Healthcare Benefits

Income Replacement Programs

401(k) with Employer Match

Global Benefits Programs

Professional Development Support

Caregiving Support

Family Planning Support

Mental Health and Coaching Benefits

Job summary

Reddit, Inc. is seeking a seasoned engineer to lead the development of its agent optimization and evaluation platform.

You will oversee evaluation environments, benchmarking, offline replay, and synthetic task generation while driving production feedback loops to improve system reliability and efficiency. You will design model routing and orchestration, define productivity metrics, and examine agent behavior in collaboration with platform, infrastructure, and developer productivity teams.

Qualifications

• Requires 5-12+ years of engineering experience and a track record of building and shipping large-scale production systems.
• Experience with evaluation or measurement systems; strong knowledge of LLMs and AI agents.
• Ability to optimize quality, latency, reliability, and cost; capable of leading strategic work independently and collaboratively.

Responsibilities

• Lead development of Reddit’s agent optimization and evaluation platform including evaluation environments, benchmarking, offline replay, synthetic task generation, regression detection, and production feedback loops.
• Design model routing and orchestration; define productivity and velocity metrics; investigate agent behavior.
• Collaborate with platform, infrastructure, and developer productivity teams.

Skills

Software Engineering

AI Agent Optimization

AI Evaluation

Benchmarking

Large Language Models

Model Orchestration

Model Routing

Distributed Systems

Production Systems

Experimentation

Regression Detection

Synthetic Task Generation

Offline Replay

Human-in-the-Loop Feedback

Developer Productivity

Cross-Functional Collaboration

Job description

Reddit, Inc. is seeking a seasoned engineer to lead the development of its agent optimization and evaluation platform.

You will oversee evaluation environments, benchmarking, offline replay, and synthetic task generation while driving production feedback loops to improve system reliability and efficiency. You will design model routing and orchestration, define productivity metrics, and examine agent behavior in collaboration with platform, infrastructure, and developer productivity teams.