Founding AI Inference Architect

General Compute; San Francisco (CA); On-site; USD 180,000 - 320,000; Full time

Job summary

General Compute is building a pioneering neocloud for alternative chips to accelerate AI inference. You will own the inference serving stack end-to-end, from request routing to autoscaling, while ensuring reliability under real customer load and tight latency demands.

This founding role sits at the intersection of hardware and software, demanding architectural judgment, collaboration with compiler and bring-up teams, and a relentless focus on throughput and cost-per-token reduction as the team

Qualifications

• 5+ years building and operating production systems at the infrastructure layer.
• Direct experience with LLM inference serving in production environments.
• Comfortable owning reliability and being on-call for critical systems.
• Strong systems fundamentals, including concurrency, networking, and scheduling.
• Self-directed, able to work in ambiguity as a founding engineer.

Responsibilities

• Own the inference serving stack end-to-end, including routing, batching, scheduling, and autoscaling.
• Continuously tune batching, KV-cache handling, and hardware utilization for throughput advantages.
• Build for reliability with monitoring, alerting, and failover for real customer load.
• Define interfaces between model compile and live deployment with the compiler and bring-up teams.
• Shape the serving roadmap, including multi-tenant isolation and new scheduling strategies.
• Set technical bar and influence how the serving team grows and maintains code quality.

Skills

LLM inference serving

High-throughput systems

On-call reliability

Performance optimization

Ambiguity tolerance

Tools

vLLM

TensorRT-LLM

TGI

SGLang

Job description

General Compute is building a pioneering neocloud for alternative chips to accelerate AI inference. You will own the inference serving stack end-to-end, from request routing to autoscaling, while ensuring reliability under real customer load and tight latency demands.

This founding role sits at the intersection of hardware and software, demanding architectural judgment, collaboration with compiler and bring-up teams, and a relentless focus on throughput and cost-per-token reduction as the team