Latency-Driven Infrastructure Engineer for AI Services

Zute Predictive

Omaha (NE)

Hybrid

USD 140,000 - 190,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Zute Predictive is seeking an engineer to build and operate the routing layer that directs requests among Claude, GPT, Gemini, Grok, and customer fine-tunes. You will own latency and cost visibility, implement robust fallback and retry mechanisms, and instrument all aspects of calls with per-customer cost attribution.

You will work with forward-deployed engineers to meet per-engagement SLAs, ensuring reliable integrations even during provider incidents.

Qualifications

  • Deep experience with distributed systems under real load.
  • Built or operated a service with strict latency SLAs (p95 < 100ms).
  • Understanding of LLM provider APIs and failure modes.
  • Know when to add abstractions vs. write code.
  • Comfortable on call for a critical system.

Responsibilities

  • Design and operate the routing layer that dispatches requests across Claude, GPT, Gemini, Grok, and customer fine-tunes.
  • Own latency and cost as observability signals.
  • Build the fallback and retry logic to keep integrations alive during provider incidents.
  • Instrument request traces, model call logs, and cost attribution per customer.
  • Collaborate with forward-deployed engineers to meet per-engagement SLAs.

Skills

Distributed systems
Low-latency design
On-call/operations
Observability

Tools

LLM provider APIs
Tracing & logging

Job description

Zute Predictive is seeking an engineer to build and operate the routing layer that directs requests among Claude, GPT, Gemini, Grok, and customer fine-tunes. You will own latency and cost visibility, implement robust fallback and retry mechanisms, and instrument all aspects of calls with per-customer cost attribution.

You will work with forward-deployed engineers to meet per-engagement SLAs, ensuring reliable integrations even during provider incidents.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

ZP-INF-002 Infrastructure engineer · model routing Omaha / Remote · HYBRID PLATFORM →
ZP-INF-002 Infrastructure engineer · model routing Omaha / Remote · HYBRID PLATFORM →

Zute Predictive • Omaha (NE)

Hybrid
USD 140,000 - 190,000
Infrastructure Engineer (Latency)
Infrastructure Engineer (Latency)

SellSig Inc. • Minnesota

On-site
USD 140,000 - 180,000
Remote LATAM Platform & Infrastructure Engineer for AI Infra
Remote LATAM Platform & Infrastructure Engineer for AI Infra

Prediktive • Atlanta (GA)

Remote
USD 120,000 - 180,000
Real-Time ML Inference Lead - Low-Latency Architect
Real-Time ML Inference Lead - Low-Latency Architect

NTT DATA, Inc. • Charlotte (NC)

On-site
USD 166,000 - 185,000
Lead Inference Systems Engineer - AI Serving
Lead Inference Systems Engineer - AI Serving

adaption • United States

Hybrid
USD 180,000 - 280,000
Flexible work
Travel stipend
Lunch stipend
+1
Senior Backend Engineer - AI Inference & Low-Latency
Senior Backend Engineer - AI Inference & Low-Latency

Fdetimes • San Francisco (CA)

On-site
USD 200,000 - 290,000
Startup equity
Health insurance
Backend Engineer: AI APIs & Global STT (Go) - Hybrid
Backend Engineer: AI APIs & Global STT (Go) - Hybrid

Plaud • East Palo Alto (CA)

Hybrid
USD 140,000 - 219,000
Founding team
Performance bonus
Equity
+3
Senior Backend & AI Platform Engineer (Low-Latency)
Senior Backend & AI Platform Engineer (Low-Latency)

Pokee AI • United States

Remote
USD 120,000 - 180,000
Senior AI Gateway Engineer - Low-Latency Backend
Senior AI Gateway Engineer - Low-Latency Backend

The Briefed • San Francisco (CA)

On-site
USD 196,000 - 294,000
Market-leading compensation
Equity compensation
Work-from-home setup budget
+1
Infrastructure Engineer: AI Backbone & Cloud Automation
Infrastructure Engineer: AI Backbone & Cloud Automation

Netic • San Francisco (CA)

On-site
USD 140,000 - 200,000