Senior Staff Engineer - AI ADC

Insight Global; San Jose (CA); Hybrid; USD 180,000 - 195,000; Full time

Job summary

Insight Global is hiring a Senior Staff Engineer - AI ADC to lead the architecture and delivery of AI‑driven application delivery components. The role blends ADC/SLB expertise with practical AI/LLM system experience to enable intelligent routing and inference‑aware load balancing across complex enterprise environments.

The candidate will drive architecture, design, prototyping, and execution, collaborating with product management, platform teams, and AI strategy groups in a hybrid on‑site

Qualifications

• Advanced degree in Computer Science, Networking, or related field (MS/PhD preferred).
• Extensive experience with ADCs, SLB, and GSLB architectures for AI workloads.
• Proven ability to drive architecture, design, prototyping, and execution at senior staff level.
• Strong programming skills in C/C++ (data plane) and Go/Python (control/ML).
• Experience with AI/ML systems, including model inference pipelines and LLM integrations.

Responsibilities

• Lead architecture and development of AI Gateway components for intelligent routing of LLM/AI traffic.
• Design and implement LLM‑aware load balancing incorporating semantic, token, latency, and model insights.
• Develop enhancements for ADC and GSLB platforms to support AI workloads, including token‑aware rate limiting and latency‑based routing.
• Integrate with vector databases, model registries, and ML observability systems.
• Drive system‑wide architecture: data plane, control plane, configuration, and distributed state management.

Skills

ADC architecture

SLB / DNS‑GSLB

AI/LLM systems

architecture leadership

prototyping & execution

traffic engineering

latency optimization

multi-model orchestration

Education

MS/PhD preferred in CS or Networking

Tools

eBPF

perf

VTune

flamegraphs

vector databases

model registries

ML observability systems

Job description

Job Description

A global technology client of Insight Global is seeking a Senior Staff Engineer - AI ADC to join their team. This company specializes in high-performance application delivery, network security, and AI-ready infrastructure solutions for enterprises, service providers, and government organizations.

This role combines deep domain expertise in Application Delivery Controllers (ADC), Server Load Balancing (SLB), DNS/GSLB, with practical experience in AI/LLM systems, to build intelligent routing, inference‑aware load balancing, and Layer‑8 contextual decision engines for AI workloads. The ideal candidate is a senior staff engineer who can drive architecture, design, prototyping, and execution—working closely with product management, platform teams, and AI strategy groups.

Key Responsibilities Include:

• Lead architecture and development of AI Gateway components for intelligent routing of LLM/AI application traffic.
• Design and implement LLM‑aware load balancing, incorporating semantic, token, latency, and model‑level insights into traffic decisions.
• Develop enhancements for ADC and GSLB platforms to support AI workloads, including:
  • Token‑aware rate limiting
  • Inference latency‑based routing
  • AI model endpoint discovery and health checks
  • Integration with vector databases, model registries, and ML observability systems
• Contribute to advanced features including Layer‑8 context‑aware routing, adaptive traffic shaping, and AI‑driven anomaly detection.
• Drive system‑wide architecture: data plane, control plane, configuration, and distributed state management.

This role is permanent and will be worked hybrid on-site in San Jose, CA. We are looking to pay between $180,000-$195,000 base salary.

Skills and Requirements

• Advanced degree in Computer Science, Networking, or related field (MS/PhD preferred)
• Deep knowledge of L4–L7 protocols (TCP/TLS/HTTP/2/3, QUIC), load balancing algorithms and GSLB strategies (DNS‑based, HTTP redirect, anycast)
• Strong programming skills in C/C++ (data plane), Go/Python (control/ML), and performance profiling (perf, eBPF, flamegraphs, VTune/nvprof)
• Hands‑on experience with AI/ML systems, including:
  • Model inference pipelines, LLM API integrations
  • Token‑level behavior and performance characteristics
  • Understanding of AI workload routing challenges (latency, caching, batching, multi‑model orchestration)

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [e-mail].