Staff AI Engineer: Production Agent Platform Leader

Cobalt; Boston (MA); On-site; USD 275,000 - 300,000; Full time

Benefits offered by this job

401(k) program

Medical, dental, vision and life保险

Equity plan

Paid time off

Parental leave

Work-from-home stipend

Job summary

Cobalt is seeking a Staff AI Engineer to lead production agent systems and improve reliability, security, and cost efficiency. You will set technical direction for building and operating the agent platform, while staying hands-on to strengthen foundations used across teams.

You’ll mentor engineers, collaborate with product, research and security, and drive practical standards for evaluation, observability and safe tool execution.

Qualifications

• 8–10+ years of software engineering leadership.
• Hands-on experience building and operating production LLM-based agents.
• Proven record improving agent reliability and reducing failures.
• Deep knowledge of agent architectures: orchestration, context, memory management.
• Experience with cloud model platforms, APIs, and cost tracking.
• Strong backend and distributed systems fundamentals.
• Experience with production observability and incident investigations.
• Understanding of security risks in agent systems.

Responsibilities

• Set technical direction for agent systems across teams, including orchestration, tool use, context and memory management, human oversight and failure recovery.
• Design, build and operate shared foundations: evaluation frameworks, agent observability and tracing, execution controls, and model access with routing and failover where needed.
• Establish evaluation practices that measure task completion, accuracy, false positives, security and cost per successful task.
• Make agent workflows resilient to tool failures, timeouts, rate limits and partial execution, with clear limits on retries, runtime and spend.
• Work with security and platform teams to build scoped permissions, data isolation, audit trails and safe execution of tools and code.
• Establish practical engineering standards: evals in CI, prompt and model versioning, regression testing for non-deterministic systems, cost and latency budgets, staged rollouts with rollback paths.
• Lead model selection and cost optimization. Benchmark models against real workloads and trade-offs between quality, latency and cost.
• Partner with teams on production readiness, incident investigation and improvements to agent reliability.
• Help teams use AI coding agents across planning, implementation, review and testing to improve delivery speed and software quality.
• Mentor engineers and technical leads through design reviews, pairing, internal talks and guidance.
• Turn business priorities into a technical roadmap with clear outcomes.

Skills

Software engineering

LLM-based agents

Agent reliability

Distributed systems

Security awareness

Tools

GCP

Go

Kubernetes

Job description

Cobalt is seeking a Staff AI Engineer to lead production agent systems and improve reliability, security, and cost efficiency. You will set technical direction for building and operating the agent platform, while staying hands-on to strengthen foundations used across teams.

You’ll mentor engineers, collaborate with product, research and security, and drive practical standards for evaluation, observability and safe tool execution.