Founding Inference Engineer - AI Serving Lead

General Compute

San Francisco (CA)

On-site

USD 190,000 - 230,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

General Compute is building neocloud for alternative chips. You’ll own the inference serving stack that sits between bought models and live customer requests: request routing, batching, KV-cache management, autoscaling across our ASIC fleet.

This is a founding role on a small team, with no separate SRE org or platform team, so you’ll design the serving architecture and be paged when it breaks. You’ll shape the roadmap and set the technical bar for the team.

Qualifications

  • 5+ years building and operating production systems at the infrastructure layer.
  • Direct experience with LLM inference serving in production.
  • Comfortable owning reliability and on-call responsibilities.
  • Strong systems fundamentals including concurrency, networking, and scheduling.
  • Self-directed and comfortable with ambiguity in a founding role.

Responsibilities

  • Own the inference serving stack end-to-end, including routing, batching, scheduling, and autoscaling across our ASIC fleet.
  • Push cost-per-token down by tuning batching, KV-cache handling, and hardware utilization.
  • Build for reliability from day one with monitoring, alerting, and failover for real customer load.
  • Define interfaces between model compilation and live serving with the bring-up team.
  • Shape the roadmap for serving features beyond the backlog, guiding the team.
  • Set the technical bar as a founding engineer to influence how the serving team grows.

Skills

LLM inference serving
High-throughput / low-latency systems
On-call reliability
Distributed systems
Python / systems programming

Tools

vLLM
TensorRT-LLM
SGLang
C++/Rust

Job description

General Compute is building neocloud for alternative chips. You’ll own the inference serving stack that sits between bought models and live customer requests: request routing, batching, KV-cache management, autoscaling across our ASIC fleet.

This is a founding role on a small team, with no separate SRE org or platform team, so you’ll design the serving architecture and be paged when it breaks. You’ll shape the roadmap and set the technical bar for the team.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Founding Platform Engineer: AI Inference Cloud
Founding Platform Engineer: AI Inference Cloud

General Compute • San Francisco (CA)

On-site
USD 240,000 - 300,000
Founding Inference Serving Engineer
Founding Inference Serving Engineer

Black.ai • San Francisco (CA)

On-site
USD 190,000 - 240,000
Founding Inference Engineer
Founding Inference Engineer

General Compute • San Francisco (CA)

On-site
USD 190,000 - 230,000
Head of AI Inference Infrastructure
Head of AI Inference Infrastructure

Black.ai • San Francisco (CA)

On-site
USD 180,000 - 260,000
Inference Engineer
Inference Engineer

Black.ai • San Francisco (CA)

On-site
USD 190,000 - 240,000
Founding Platform Engineer
Founding Platform Engineer

General Compute • San Francisco (CA)

On-site
USD 240,000 - 300,000
Founding Network Engineer for AI Inference Data Center
Founding Network Engineer for AI Inference Data Center

General Compute • San Francisco (CA)

On-site
USD 180,000 - 260,000
Platform Engineer
Platform Engineer

Black.ai • San Francisco (CA)

On-site
USD 180,000 - 260,000
Lead Inference Engineer — AI Serving & GPU Optimization
Lead Inference Engineer — AI Serving & GPU Optimization

Aleroy • United States

Remote
USD 140,000 - 190,000
Head of Inference Cloud Infrastructure
Head of Inference Cloud Infrastructure

General Compute • San Francisco (CA)

Remote
USD 180,000 - 250,000