Software Engineer, Agent Optimization & Evaluation

Ladders; United States; Remote; USD 110,000 - 325,000; Full time

Benefits offered by this job

Healthcare and income replacement

401(k) with employer match

Global lifestyle support

Family planning support

Gender-affirming care

Mental health and coaching

Flexible vacation and paid volunteer

Paid parental leave

Job summary

Ladders is seeking a Software Engineer, Agent Optimization & Evaluation to join our team to build the infrastructure for evaluating and improving AI agent performance at scale. You will help shape systems that measure quality, latency, reliability, and cost while enabling data-driven decisions about routing, benchmarking, and regression detection.

The role partners with cross-functional teams to integrate agent capabilities and support broader product and engineering goals.

Qualifications

• 5-12+ years of engineering experience.
• Extensive experience in large-scale production software systems.
• Experience designing evaluation and benchmarking systems for AI or complex distributed systems.
• Strong understanding of modern LLMs and AI agents.
• Proven ability to optimize systems for quality, latency, reliability, and cost.
• Excellent software engineering fundamentals.
• Strong cross-functional collaboration skills.

Responsibilities

• Design and build the agent optimization harness.
• Develop evaluation environments to measure agent performance.
• Build systems for automated evaluation and regression detection.
• Design model routing strategies for optimization.
• Define metrics to measure developer productivity.
• Build tooling for analyzing agent behavior.
• Collaborate with cross-functional teams to integrate agent capabilities.

Skills

Large-scale systems

AI/LLMs

Evaluation benchmarks

Software engineering

Cross-functional collaboration

Performance optimization

Job description

For our client, we are seeking a Software Engineer, Agent Optimization & Evaluation to join the team of a leader in the Media space.

This role will focus on building the infrastructure and tooling needed to evaluate and improve AI agent performance at scale. You will help shape systems that measure quality, latency, reliability, and cost, while enabling data-driven decisions about routing, benchmarking, and regression detection. The position partners closely with cross-functional teams to integrate agent capabilities and support broader product and engineering goals. It is a strong fit for an engineer who enjoys combining large-scale systems work with applied AI evaluation.

Location: Remote - US based candidates only, no visa sponsorship available

Compensation: $109,800 - $325,000 annually

Responsibilities

• Design and build the agent optimization harness
• Develop evaluation environments to measure agent performance
• Build systems for automated evaluation and regression detection
• Design model routing strategies for optimization
• Define metrics to measure developer productivity
• Build tooling for analyzing agent behavior
• Collaborate with cross-functional teams to integrate agent capabilities

Qualifications

• 5-12+ years of engineering experience
• Extensive experience in large-scale production software systems
• Experience designing evaluation and benchmarking systems for AI or complex distributed systems
• Strong understanding of modern LLMs and AI agents
• Proven ability to optimize systems for quality, latency, reliability, and cost
• Excellent software engineering fundamentals
• Strong cross-functional collaboration skills

Benefits

• Comprehensive healthcare benefits and income replacement programs
• 401(k) with employer match
• Global benefit programs for lifestyle support
• Family planning support
• Gender-affirming care
• Mental health and coaching benefits
• Flexible vacation and paid volunteer time off
• Generous paid parental leave

Our client is an equal opportunity employer.