AI Agent Software Engineer - Agent Performance Engineering

Assembled; San Francisco (CA); Hybrid; USD 150,000 - 300,000; Full time

Job summary

Assembled, a leader in customer support technology, seeks an AI Agent Software Engineer to enhance AI performance. In this pivotal role, you will develop systems that automate knowledge management and improve retrieval accuracy, directly impacting the efficiency and quality of customer interactions.

Benefits

Generous medical, dental, and vision benefits

Paid company holidays and unlimited time off

Monthly professional development credits

Paid parental leave

Catered lunches and snacks in office

Qualifications

• 5+ years of software engineering experience as an individual contributor.
• Experience with AI evaluation and data system architectures.
• Passion for optimizing AI performance and creating cutting-edge tools.

Responsibilities

• Develop evaluation frameworks to optimize AI agent performance.
• Design systems for automated knowledge generation and management.
• Enhance AI retrieval and accuracy through continuous refinement.

Skills

AI evaluation systems

Data pipelines

AI model optimization

Knowledge bases

Information extraction

Python

Golang

Education

Bachelor's degree in Computer Science or related field

Tools

LLMs

Vector databases

AI evaluation tools

Benchmarking tools

Job description

AI Agent Software Engineer - Agent Performance Engineering

Join to apply for the AI Agent Software Engineer - Agent Performance Engineering role at Assembled

AI Agent Software Engineer - Agent Performance Engineering

Join to apply for the AI Agent Software Engineer - Agent Performance Engineering role at Assembled

Assembled builds the infrastructure that underpins exceptional customer support, empowering companies like CashApp, Etsy, and Robinhood to deliver faster, better service at scale. With solutions for workforce management, BPO collaboration, and AI-powered issue resolution, Assembled simplifies the complexities of modern support operations by uniting in-house, outsourced, and AI-powered agents in a single operating system. Backed by $70M in funding from NEA, Emergence Capital, and Stripe, and driven by a team of experts passionate about problem-solving, we’re at the forefront of support operations technology.

The Team

Our Agent Intelligence team is focused on maximizing AI agent performance and driving volume automation across voice, chat, and email channels. This team is vital for our vision of creating the ultimate omnichannel AI customer support agent and represents one of the fastest growing areas both inside our company and across the industry.

We're currently one of the largest companies deploying AI agents in production at massive scale, working with enterprise clients like Canva, Etsy, and Ramp. The engineering challenges are significant and largely unsolved: building sophisticated evaluation systems, automating knowledge generation, optimizing AI agent accuracy, and creating the infrastructure needed to continuously improve agent intelligence at scale.

Your Impact

As part of the Agent Performance Engineering team, you'll be working on the core systems that make our AI agents smarter, more accurate, and more capable of handling complex customer interactions. You'll be building the evaluation frameworks, knowledge automation systems, and intelligence optimization tools that directly impact our ability to automate customer support at enterprise scale.

Responsibilities

• Build foundational evaluation infrastructure: Develop comprehensive evaluation systems from the ground up, including golden dataset creation, automated benchmarking, and model comparison tools. You'll help create the frameworks that enable us to measure and optimize AI agent performance across all communication channels.
• Automate knowledge generation: Design and implement systems that automatically create synthetic guides, documentation, and metadata to improve agent knowledge bases. You'll work on cutting-edge approaches to knowledge extraction and augmentation.
• Optimize AI agent accuracy: Enhance our retrieval systems, implement advanced prompt optimization techniques, and build tools that continuously improve agent responses through automated evaluation and refinement.
• Develop intelligence infrastructure: Architect systems that enable rapid model upgrades, A/B testing of different AI approaches, and scalable evaluation pipelines that support enterprise deployment.
• Drive volume automation: Focus on the north star goal of maximizing automated resolutions across voice, chat, and email by building the intelligence systems that make it possible.

About You

You might be a good fit if you:

• Have 5+ years of experience in software engineering as an individual contributor
• Have experience with AI evaluation systems, data pipelines, or AI model optimization
• Are passionate about building systems that measure and improve AI performance
• Have worked with retrieval systems, knowledge bases, or information extraction
• Enjoy building tools and infrastructure that enable other engineers and AI systems to perform better
• Are highly ambitious and driven, setting high goals for yourself and others
• Put customers first, focusing on solving real problems that impact support quality
• Enjoy fast-paced environments and can quickly adjust when new insights emerge
• Have a bit of a maverick streak that helps you come up with creative solutions
• Have made a noticeable impact on small teams and have solid experience in startups or smaller companies
• Stay humble and open to feedback, value teamwork, and are always ready to learn and grow

Technologies You'll Work With

• Python/Golang for evaluation systems and data processing
• LLMs and prompt optimization frameworks
• Vector databases and retrieval systems
• AI evaluation and benchmarking tools
• Data pipeline and automation infrastructure

The estimated base salary range for this role is $150,000 - $300,000 per year. The base pay offered may vary depending on location, job-related knowledge, skills, and experience. Stock options are provided as part of the compensation package, in addition to a full range of medical, financial, and/or other benefits, dependent on the position offered.

• Generous medical, dental, and vision benefits
• Paid company holidays, sick time, and unlimited time off
• Monthly credits to spend on each: professional development, general wellness, Assembled customers, and commuting
• Paid parental leave
• Hybrid work model with catered lunches everyday (M-F), snacks, and beverages in our SF & NY offices

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Engineering and Information Technology

• Industries

  Software Development

Referrals increase your chances of interviewing at Assembled by 2x

Get notified about new Software Engineer jobs in San Francisco, CA.

San Francisco, CA $145,000.00-$250,000.00 3 days ago

San Francisco, CA $130,000.00-$238,000.00 2 weeks ago

Software Engineer Frontend (Multiple Levels) - Slack

San Francisco, CA $172,000.00-$334,600.00 1 day ago

San Francisco, CA $150,000.00-$230,000.00 1 month ago

San Francisco, CA $170,000.00-$270,000.00 2 days ago

San Francisco, CA $125,000.00-$175,000.00 4 days ago

San Francisco, CA $150,000.00-$176,000.00 1 month ago

Alameda, CA $130,000.00-$160,000.00 3 days ago

San Francisco, CA $130,000.00-$190,000.00 2 weeks ago

San Francisco, CA $130,000.00-$240,000.00 3 days ago

San Francisco, CA $120,000.00-$190,000.00 7 months ago

San Francisco, CA $35.00-$40.00 19 hours ago

Software Engineer, Frontend (All Levels)

San Francisco, CA $150,000.00-$220,000.00 2 weeks ago

San Francisco, CA $112,000.00-$160,000.00 2 weeks ago

San Francisco, CA $120,000.00-$200,000.00 2 years ago

San Francisco, CA $105,600.00-$198,000.00 2 weeks ago

San Francisco, CA $130,000.00-$185,000.00 1 month ago

San Francisco, CA $165,000.00-$165,000.00 2 years ago

San Francisco, CA $140,000.00-$280,000.00 6 months ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.