Software Engineer, Agent Optimization & Evaluation

Reddit, Inc.; San Francisco (CA); On-site; USD 180,000 - 300,000; Full time

Benefits offered by this job

Comprehensive Healthcare Benefits

Income Replacement Programs

401(k) with Employer Match

Global Benefits Programs

Professional Development Support

Caregiving Support

Family Planning Support

Mental Health and Coaching Benefits

Job summary

Reddit, Inc. is seeking a seasoned engineer to lead the development of its agent optimization and evaluation platform.

You will oversee evaluation environments, benchmarking, offline replay, and synthetic task generation while driving production feedback loops to improve system reliability and efficiency. You will design model routing and orchestration, define productivity metrics, and examine agent behavior in collaboration with platform, infrastructure, and developer productivity teams.

Qualifications

• Requires 5-12+ years of engineering experience and a track record of building and shipping large-scale production systems.
• Experience with evaluation or measurement systems; strong knowledge of LLMs and AI agents.
• Ability to optimize quality, latency, reliability, and cost; capable of leading strategic work independently and collaboratively.

Responsibilities

• Lead development of Reddit’s agent optimization and evaluation platform including evaluation environments, benchmarking, offline replay, synthetic task generation, regression detection, and production feedback loops.
• Design model routing and orchestration; define productivity and velocity metrics; investigate agent behavior.
• Collaborate with platform, infrastructure, and developer productivity teams.

Skills

Software Engineering

AI Agent Optimization

AI Evaluation

Benchmarking

Large Language Models

Model Orchestration

Model Routing

Distributed Systems

Production Systems

Experimentation

Regression Detection

Synthetic Task Generation

Offline Replay

Human-in-the-Loop Feedback

Developer Productivity

Cross-Functional Collaboration

Job description

Lead development of Reddit’s agent optimization and evaluation platform, including evaluation environments, benchmarking, offline replay, synthetic task generation, regression detection, and production feedback loops. Design model routing and orchestration, define productivity and velocity metrics, investigate agent behavior, and collaborate with platform, infrastructure, and developer productivity teams.

Requirements

Requires 5-12+ years of engineering experience and a record of building and shipping large-scale production systems. Candidates should have experience with evaluation or measurement systems, strong knowledge of LLMs and AI agents, the ability to optimize quality, latency, reliability, and cost, and the independence and collaboration skills to lead strategic work.

Key Skills

• Software Engineering
• AI Agent Optimization
• AI Evaluation
• Benchmarking
• Large Language Models
• Model Orchestration
• Model Routing
• Distributed Systems
• Production Systems
• Experimentation
• Regression Detection
• Synthetic Task Generation
• Offline Replay
• Human-in-the-Loop Feedback
• Developer Productivity
• Cross-Functional Collaboration

Benefits

• Comprehensive Healthcare Benefits
• Income Replacement Programs
• 401(k) With Employer Match
• Global Benefits Programs
• Professional Development Support
• Caregiving Support
• Family Planning Support
• Gender-Affirming Care
• Mental Health and Coaching Benefits
• Flexible Vacation
• Paid Volunteer Time Off
• Paid Parental Leave
• Equity in the Form of Restricted Stock Units