Inference Engineer

Acceler8 Talent; California; On-site; USD 210,000 - 320,000; Full time

Benefits offered by this job

Vision insurance

Medical insurance

Dental insurance

401(k)

Job summary

Acceler8 Talent is seeking a Founding Engineer to design and build scalable LLM infrastructure and inference systems in Mountain View, CA. You will own architecture and implementation, collaborating directly with the founding team to shape the core technology from day one.

This hands-on role focuses on distributed serving, performance optimization, and system-level engineering across compute, memory, and network boundaries.

Qualifications

• Hands-on experience building or optimizing LLM inference infrastructure.
• Strong understanding of distributed systems and multi-machine execution.
• Ability to identify performance bottlenecks and implement measurable improvements.

Responsibilities

• Design, build, and optimize scalable infrastructure for hosting and serving large language models across multiple machines.
• Own architecture and implementation of distributed LLM inference systems with technical autonomy.
• Develop infrastructure supporting agentic AI workloads and evolving inference requirements.
• Analyze system performance across compute, memory, and distributed execution.
• Investigate bottlenecks affecting inference latency, throughput, hardware utilization, and efficiency.
• Optimize inference serving strategies including batching, scheduling, caching, and parallel execution.
• Develop benchmarking methodologies for representative LLM workloads.
• Collaborate with performance and infra engineers to translate workload optimizations into system-wide improvements.
• Work with founding team to influence architecture and platform development.

Skills

LLM inference engineering

Distributed systems

Performance optimization

Multi-machine execution

Debugging complex systems

Architectural design

Model serving frameworks

Education

BS in Computer Science or equivalent

Tools

vLLM

TensorRT-LLM

DeepSpeed

Megatron

Job description

Founding Engineer – LLM Infrastructure & Inference

Location:Mountain View / South Bay, CA

Work Arrangement:Onsite

About the Role

An innovative AI infrastructure startup is seeking a Founding Engineer specializing in LLM infrastructure and inference to help build next-generation AI serving systems.

The company is developing intelligent software that analyzes AI training and inference workloads to improve performance, power efficiency, and infrastructure costs.

This is a highly hands‑on engineering opportunity focused on designing, building, and optimizing distributed LLM serving infrastructure for large language models and agentic AI workloads.

The founding engineer will take significant ownership of system architecture and implementation, working directly with the founding team to develop the company's core technology. Opportunities are also available for exceptional early‑career engineers with strong computer science fundamentals and demonstrated technical ability.

Key Responsibilities

• Design, build, and optimize scalable infrastructure for hosting and serving large language models across multiple machines.
• Own architecture and implementation of distributed LLM inference systems, with significant technical autonomy.
• Develop infrastructure supporting agentic AI workloads and evolving inference requirements.
• Analyze system performance across compute, memory, and distributed execution.
• Investigate bottlenecks affecting inference latency, throughput, hardware utilization, and overall efficiency.
• Optimize inference serving strategies, including batching, scheduling, caching, and parallel execution.
• Develop performance measurement and benchmarking methodologies for representative LLM workloads.
• Evaluate and improve KV cache management, memory utilization, and inference execution efficiency.
• Collaborate with performance and infrastructure engineers to translate workload‑level optimizations into system‑wide improvements.
• Work directly with the founding team to influence architecture, engineering decisions, and development of the serving platform.

Required Qualifications

• Hands‑on experience building, implementing, or optimizing LLM inference or model‑serving infrastructure.
• Strong understanding of distributed systems and multi‑machine execution.
• Understanding of how model execution interacts with compute, memory, and machine‑level constraints.
• Ability to identify performance bottlenecks and implement measurable improvements in latency, throughput, or hardware utilization.
• Strong software engineering fundamentals and experience debugging complex systems.
• Ability to explain architectural decisions, technical trade‑offs, and measured performance improvements.
• Willingness to work across infrastructure, inference engines, runtime behavior, and system optimization in an early‑stage engineering environment.
• For founding‑level candidates, demonstrated ability to independently design and build substantial systems.

For Junior Candidates:

Strong computer science fundamentals, substantial engineering projects, and demonstrated ability to learn complex technical concepts quickly.

Preferred Qualifications

• Experience developing or contributing to LLM serving frameworks such as vLLM, SGLang, TensorRT‑LLM, or llm‑d.
• Understanding of inference optimization techniques, including KV cache management, continuous batching, PagedAttention, and FlashAttention.
• Experience optimizing inference latency, including time to first token (TTFT), throughput, and serving efficiency.
• Familiarity with distributed execution techniques such as tensor parallelism, pipeline parallelism, and expert parallelism.
• Experience with distributed machine learning frameworks such as DeepSpeed, Megatron, or FSDP.
• Knowledge of GPU performance profiling, memory management, and hardware utilization.
• Experience with agentic AI workloads and dynamic inference demands.
• Background in high‑performance computing, distributed systems, or AI infrastructure.

Why Join?

• Take foundational ownership of LLM serving infrastructure and distributed inference systems.
• Work directly with an experienced founding team with backgrounds in advanced AI hardware and accelerator development.
• Influence the architecture and technical direction of an emerging AI infrastructure platform.
• Build infrastructure supporting LLM and agentic AI workloads.
• Solve challenging performance problems spanning distributed systems, memory management, and inference optimization.
• Join a small engineering team where individual contributions directly influence product development.
• Opportunities for both experienced founding‑level engineers and exceptional junior engineers.

The team values strong engineering fundamentals, technical curiosity, independent problem‑solving, and the ability to explain complex systems and performance improvements in detail.

Featured benefits

Vision insurance, Medical insurance, Dental insurance, 401(k)