Senior Staff AI/ML Inference Engineer - GPU, Kubernetes

ServiceNow; Santa Clara (CA); Hybrid; USD 201,000 - 352,000; Full time

Benefits offered by this job

Health plans

401(k) Plan with company match

ESP P (Employee Stock Purchase Plan)

Matching donations

Flexible time away plan

Family leave programs

Job summary

ServiceNow is seeking a Senior Staff Software Engineer - AI/ML Inference Engineering to lead architecture, optimization, and production readiness of our AI inference platform. You will own the end-to-end lifecycle across multiple backends on Kubernetes and design gateways with Envoy for secure, observable, and scalable AI workloads.

The role requires deep expertise in GPU compute, LLM APIs, and modern inference frameworks, with a focus on pragmatic engineering and measurable performance impact.

Qualifications

• 8+ years of Backend software engineering experience.
• 4+ years of designing and operating distributed systems.
• Bachelor’s degree in computer science, software engineering, or a closely related technical field.
• A master’s degree may offset up to one year of the experience minimum.

Responsibilities

• Lead the architecture, optimization, and operational excellence of the AI inference platform.
• Own the end-to-end inference lifecycle from onboarding models to deploying optimized engines on Kubernetes.
• Architect the intelligent gateway layer using Envoy and ext_proc for access control, observability, and routing.

Skills

Go or Python

OOP

Design patterns

Distributed systems design

Model serving

Education

Bachelor’s degree in computer science or related field

Master’s degree offset experience

Tools

vLLM

TensorRT-LLM

Triton Inference Server

Envoy

Job description

ServiceNow is seeking a Senior Staff Software Engineer - AI/ML Inference Engineering to lead architecture, optimization, and production readiness of our AI inference platform. You will own the end-to-end lifecycle across multiple backends on Kubernetes and design gateways with Envoy for secure, observable, and scalable AI workloads.

The role requires deep expertise in GPU compute, LLM APIs, and modern inference frameworks, with a focus on pragmatic engineering and measurable performance impact.