ZP-INF-002 Infrastructure engineer · model routing Omaha / Remote · HYBRID PLATFORM →

Zute Predictive

Omaha (NE)

Hybrid

USD 140,000 - 190,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Zute Predictive is seeking an engineer to build and operate the routing layer that directs requests among Claude, GPT, Gemini, Grok, and customer fine-tunes. You will own latency and cost visibility, implement robust fallback and retry mechanisms, and instrument all aspects of calls with per-customer cost attribution.

You will work with forward-deployed engineers to meet per-engagement SLAs, ensuring reliable integrations even during provider incidents.

Qualifications

  • Deep experience with distributed systems under real load.
  • Built or operated a service with strict latency SLAs (p95 < 100ms).
  • Understanding of LLM provider APIs and failure modes.
  • Know when to add abstractions vs. write code.
  • Comfortable on call for a critical system.

Responsibilities

  • Design and operate the routing layer that dispatches requests across Claude, GPT, Gemini, Grok, and customer fine-tunes.
  • Own latency and cost as observability signals.
  • Build the fallback and retry logic to keep integrations alive during provider incidents.
  • Instrument request traces, model call logs, and cost attribution per customer.
  • Collaborate with forward-deployed engineers to meet per-engagement SLAs.

Skills

Distributed systems
Low-latency design
On-call/operations
Observability

Tools

LLM provider APIs
Tracing & logging

Job description

Build the layer that decides which model answers which request, at what latency, for what cost. Runs at the critical path of every customer integration.

WHAT YOU'LL DO
  • Design and operate the routing layer that dispatches requests across Claude, GPT, Gemini, Grok, and customer fine-tunes.
  • Own latency and cost as first-class observability signals -- not afterthoughts.
  • Build the fallback and retry logic that keeps integrations alive when a provider has an incident.
  • Instrument everything: request traces, model call logs, cost attribution per customer.
  • Work directly with forward-deployed engineers to meet per-engagement SLAs.
WHAT WE'RE LOOKING FOR
  • Deep experience with distributed systems under real load -- not demo load.
  • Have built or operated a service with strict latency SLAs (p95 < 100ms class).
  • Understand LLM provider APIs well enough to know where the failure modes live.
  • Strong instinct for when to add abstractions and when to just write the code.
  • Comfortable on call for a system that other engineers depend on to ship.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Latency-Driven Infrastructure Engineer for AI Services
Latency-Driven Infrastructure Engineer for AI Services

Zute Predictive • Omaha (NE)

Hybrid
USD 140,000 - 190,000
Software Development Engineer, CloudFront Routing
Software Development Engineer, CloudFront Routing

Amazon Inc. • Seattle (WA)

On-site
USD 150,000 - 190,000
Principal Network Engineer
Principal Network Engineer

Request Technology, LLC • Chicago (IL)

Hybrid
USD 150,000 - 190,000
Staff Software Engineer - Integrated Network Strategy
Staff Software Engineer - Integrated Network Strategy

Cloudflare, Inc. • United States

On-site
USD 180,000 - 260,000
Engineering Manager, Provider Ecosystem Remote (US)
Engineering Manager, Provider Ecosystem Remote (US)

OpenRouter, Inc • United States

Hybrid
USD 180,000 - 250,000
Restaurant example not provided
Engineering Manager, Provider Ecosystem & Multimodal Inference
Engineering Manager, Provider Ecosystem & Multimodal Inference

OpenRouter, Inc • United States

Hybrid
USD 180,000 - 250,000
Restaurant example not provided
Engineering Manager, Provider Ecosystem
Engineering Manager, Provider Ecosystem

OpenRouter, Inc • New York (NY)

On-site
USD 180,000 - 260,000
Senior Platform Engineer (Java & Python)
Senior Platform Engineer (Java & Python)

Intempt Technologies, LLC. • United States

On-site
USD 150,000 - 190,000
Founding LLM Infra Engineer — Scale Platform Reliability
Founding LLM Infra Engineer — Scale Platform Reliability

Pax Historia • San Francisco (CA)

On-site
USD 180,000 - 260,000
Equity
Visa sponsorship
Full-Stack Platform Engineer
Full-Stack Platform Engineer

SunAlgo • United States

Remote
USD 140,000 - 180,000