Senior AI/ML Inference Engineer — GPU & LLMs on Kubernetes

ServiceNow; San Diego (CA); Hybrid; USD 181,000 - 317,000; Full time

Job summary

ServiceNow in San Diego, California seeks a Senior Staff Software Engineer - AI/ML Inference Engineering to lead the architecture, optimization, and operations of our AI inference platform.

You will own the end-to-end inference lifecycle—from onboarding cutting-edge models to deploying optimized inference engines on Kubernetes backends, and you will design the intelligent gateway layer with Envoy to enforce access, provide observability, and enable advanced request routing.

Qualifications

• 8+ years of Backend software engineering experience.
• 4+ years designing and operating distributed systems.
• Bachelor’s degree in computer science or software engineering required.
• Knowledge of AI/ML techniques and algorithms is essential.

Responsibilities

• Lead architecture, optimization, and operations of the AI inference platform.
• Own end-to-end inference lifecycle—from onboarding models to production deployment on Kubernetes.
• Architect the gateway layer using Envoy/ext_proc for access control, observability, and routing.

Skills

Go

Python

OOP

Design Patterns

Algorithms

ML Serving

Education

Bachelor's degree in CS/Software Engineering

Tools

vLLM

TensorRT-LLM

Triton Inference Server

Job description

ServiceNow in San Diego, California seeks a Senior Staff Software Engineer - AI/ML Inference Engineering to lead the architecture, optimization, and operations of our AI inference platform.

You will own the end-to-end inference lifecycle—from onboarding cutting-edge models to deploying optimized inference engines on Kubernetes backends, and you will design the intelligent gateway layer with Envoy to enforce access, provide observability, and enable advanced request routing.