Staff AI Engineer - Production Agent Platform

Cobalt; United States; Remote; USD 275,000 - 300,000; Full time

Benefits offered by this job

Equity plan

401(k) program (US)

Medical, dental, vision insurance

Job summary

Cobalt is seeking a Staff AI Engineer to lead production agent systems and set the technical direction for reliable, secure, and cost-efficient AI agents. You will mentor teams, design foundations for evaluation, observability and safe execution, and partner across engineering, product, research and security.

Your focus is the agent platform, production practices, and improving how teams build software with AI coding agents. This role blends hands-on work with strategic leadership.

Qualifications

• 8–10+ years of software engineering with leadership across teams.
• Hands-on experience building and operating LLM-based agents in production.
• Proven track record improving agent reliability and reducing failures.
• Strong knowledge of agent architectures: tool use, orchestration, context and memory.
• Experience running LLM workloads on managed cloud platforms or provider APIs.
• Strong backend and distributed systems fundamentals: APIs, asynchronous workflows, concurrency.
• Experience with production observability and incident investigation.
• Understanding security risks in agent systems and data isolation.
• Evidence of impact beyond code: shared platforms, standards, mentoring.

Responsibilities

• Set technical direction for agent systems across teams.
• Design, build and operate shared foundations: evaluation frameworks, agent observability and tracing, execution controls, and model access with routing and failover where needed.
• Establish evaluation practices that measure task completion, accuracy, security and cost per successful task; work with domain experts to build test cases.
• Make agent workflows resilient to tool failures, timeouts, rate limits and partial execution, with clear limits on retries and spend.
• Work with security and platform teams to build scoped permissions, data isolation, audit trails and safe execution of tools.
• Establish engineering standards: CI evals, prompt/model versioning, regression testing, cost and latency budgets, staged rollouts with rollback paths.
• Lead model selection and cost optimization; benchmark models against real workloads and trade-offs.
• Partner with teams on production readiness, incident investigation and improvements to agent reliability.
• Help teams use AI coding agents across planning, implementation, review and testing; measure impact on delivery speed and software quality.
• Mentor engineers and technical leads through design reviews and guidance.
• Work with product/research/security to turn business priorities into a technical roadmap.

Skills

Leadership

LLM agents in production

Agent architectures

Backend & distributed systems

Observability & incident response

Security in agent systems

Cost optimization

Tools

Go

Kubernetes

GCP

Job description

Cobalt is seeking a Staff AI Engineer to lead production agent systems and set the technical direction for reliable, secure, and cost-efficient AI agents. You will mentor teams, design foundations for evaluation, observability and safe execution, and partner across engineering, product, research and security.

Your focus is the agent platform, production practices, and improving how teams build software with AI coding agents. This role blends hands-on work with strategic leadership.