Principal Engineer, Agent Platform

The Gap, Inc.; San Francisco (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

The Gap, Inc. is building an enterprise AI agent platform to enable teams across the company to create, deploy, govern, and operate AI-powered solutions at scale.

As a Principal Engineer, you will define the technical foundation for agent development, building services, frameworks, and guardrails for secure, reliable, and scalable AI agents. This hands-on role focuses on architecture, software design, and platform development across AI, distributed systems, cloud infrastructure, security, and

Qualifications

• Extensive experience building and operating large-scale distributed systems in production.
• Expert-level Python development skills with hands-on engineering focus.
• Hands-on experience delivering production LLM, generative AI, or agent-based solutions.
• Deep understanding of agent architectures, orchestration patterns, and enterprise AI platforms.
• Experience with retrieval systems, embeddings, vector search, and context management strategies.
• Knowledgeable in AI evaluation, testing methodologies, and QA for non-deterministic systems.
• Proficient with Google Cloud services including Vertex AI, Gemini, GKE, Cloud Run, BigQuery, and Pub/Sub.

Responsibilities

• Design and build core platform services enabling enterprise AI agent development and operations.
• Define architecture, patterns, and standards for the AI agent platform.
• Develop and maintain a centralized model gateway for routing, governance, and cost management.
• Build agent runtime capabilities supporting long-running workflows and human-in-the-loop processes.
• Design secure identity, authorization, and policy enforcement for agents and tools.
• Establish platform capabilities for context management, memory, retrieval, and data isolation.
• Create testing, evaluation, and release frameworks ensuring agent quality, safety, and performance.
• Implement observability, auditability, and operational tooling for monitoring agent behavior.
• Partner with Security, Data Governance, and Cloud Infrastructure teams on platform controls.
• Mentor engineers, influence direction, and raise engineering standards across teams.

Skills

Distributed systems

Python

Production LLM/agent solutions

Agent architectures

Retrieval/embeddings/vector search

Context management

AI evaluation & testing

Google Cloud (Vertex AI, Gemini, GKE,

Cloud security

Policy-as-code

Observability & reliability patterns

Influencing technical direction

Tools

Vertex AI

Gemini

GKE

Cloud Run

BigQuery

Pub/Sub

Job description

About the Role

We are building the enterprise AI agent platform that teams across the company will use to create, deploy, govern, and operate AI-powered solutions at scale. As a Principal Engineer, you will help define the technical foundation for agent development across the organization, building the services, frameworks, and guardrails that make AI agents secure, reliable, observable, and easy to adopt.

This is a hands‑on engineering role focused on architecture, software design, and platform development. You will work across AI, distributed systems, cloud infrastructure, security, and governance to deliver a platform that enables product and engineering teams to move quickly while meeting enterprise standards for security, compliance, and operational excellence.

What You’ll Do

• - Design and build core platform services that enable enterprise AI agent development and operations


• - Define the architecture, engineering patterns, and technical standards for the AI agent platform


• - Develop and maintain a centralized model gateway for routing, governance, resiliency, and cost management


• - Build agent runtime capabilities that support long-running workflows, orchestration, and human-in-the-loop processes


• - Design secure identity, authorization, and policy enforcement mechanisms for agents and tools


• - Establish platform capabilities for context management, memory, retrieval, and data isolation


• - Create testing, evaluation, and release frameworks that ensure agent quality, safety, and performance


• - Implement observability, auditability, and operational tooling for monitoring and troubleshooting agent behavior


• - Partner with Security, Data Governance, and Cloud Infrastructure teams to define platform controls and guardrails


• - Mentor engineers, influence technical direction, and raise engineering standards across multiple teams

Who You Are

• - Extensive experience building and operating large-scale distributed systems in production


• - Strong software engineering background with expert-level Python development skills


• - Hands‑on experience delivering production LLM, generative AI, or agent-based solutions


• - Deep understanding of agent architectures, orchestration patterns, and enterprise AI platforms


• - Experienced with retrieval systems, embeddings, vector search, and context management strategies


• - Knowledgeable in AI evaluation, testing methodologies, and quality assurance for non-deterministic systems


• - Proficient with Google Cloud services including Vertex AI, Gemini, GKE, Cloud Run, BigQuery, and Pub/Sub


• - Experienced with cloud security, workload identity, OAuth, policy-as-code, and secure-by-design architectures


• - Strong understanding of distributed system reliability patterns, including resiliency, retries, observability, and fault tolerance


• - Proven ability to influence technical direction, drive adoption of architectural standards, and lead through expertise rather than authority