Senior AI Agent Optimization Engineer (Remote)

Ladders; United States; Remote; USD 110,000 - 325,000; Full time

Benefits offered by this job

Healthcare and income replacement

401(k) with employer match

Global lifestyle support

Family planning support

Gender-affirming care

Mental health and coaching

Flexible vacation and paid volunteer

Paid parental leave

Job summary

Ladders is seeking a Software Engineer, Agent Optimization & Evaluation to join our team to build the infrastructure for evaluating and improving AI agent performance at scale. You will help shape systems that measure quality, latency, reliability, and cost while enabling data-driven decisions about routing, benchmarking, and regression detection.

The role partners with cross-functional teams to integrate agent capabilities and support broader product and engineering goals.

Qualifications

• 5-12+ years of engineering experience.
• Extensive experience in large-scale production software systems.
• Experience designing evaluation and benchmarking systems for AI or complex distributed systems.
• Strong understanding of modern LLMs and AI agents.
• Proven ability to optimize systems for quality, latency, reliability, and cost.
• Excellent software engineering fundamentals.
• Strong cross-functional collaboration skills.

Responsibilities

• Design and build the agent optimization harness.
• Develop evaluation environments to measure agent performance.
• Build systems for automated evaluation and regression detection.
• Design model routing strategies for optimization.
• Define metrics to measure developer productivity.
• Build tooling for analyzing agent behavior.
• Collaborate with cross-functional teams to integrate agent capabilities.

Skills

Large-scale systems

AI/LLMs

Evaluation benchmarks

Software engineering

Cross-functional collaboration

Performance optimization

Job description

Ladders is seeking a Software Engineer, Agent Optimization & Evaluation to join our team to build the infrastructure for evaluating and improving AI agent performance at scale. You will help shape systems that measure quality, latency, reliability, and cost while enabling data-driven decisions about routing, benchmarking, and regression detection.

The role partners with cross-functional teams to integrate agent capabilities and support broader product and engineering goals.