Inference Engineer

Acceler8 Talent; Palo Alto (CA); On-site; USD 180,000 - 320,000; Full time

Job summary

Acceler8 Talent is partnering with an AI infrastructure startup in Mountain View, CA to hire a Founding Engineer focused on LLM infrastructure and inference for next-generation AI serving systems.

You will own architecture, build scalable distributed inference, optimize performance, and collaborate with the founding team to shape core technology. Senior and exceptional junior engineers are encouraged to apply.

Qualifications

• Hands-on experience building, implementing, or optimizing LLM inference or model-serving infrastructure.
• Strong understanding of distributed systems and multi-machine execution.
• Ability to identify performance bottlenecks and implement measurable improvements in latency, throughput, or hardware utilization.
• For founding-level candidates, demonstrated ability to independently design and build substantial systems.

Responsibilities

• Design, build, and optimize scalable infrastructure for hosting and serving large language models across multiple machines.
• Own architecture and implementation of distributed LLM inference systems, with significant technical autonomy.
• Develop infrastructure supporting agentic AI workloads and evolving inference requirements.
• Analyze system performance across compute, memory, and distributed execution.
• Investigate bottlenecks affecting inference latency, throughput, hardware utilization, and overall efficiency.
• Optimize inference serving strategies, including batching, scheduling, caching, and parallel execution.
• Develop performance measurement and benchmarking methodologies for representative LLM workloads.
• Evaluate and improve KV cache management, memory utilization, and inference execution efficiency.
• Collaborate with performance and infrastructure engineers to translate workload-level optimizations into system-wide improvements.
• Work directly with the founding team to influence architecture, engineering decisions, and development of the serving platform.

Skills

LLM inference

Distributed systems

Performance optimization

Software engineering fundamentals

Architecture decisions

Troubleshooting complex systems

Independent design

Tools

vLLM

SGLang

TensorRT-LLM

llm-d

DeepSpeed

Megatron

FSDP

Job description

Location: Mountain View / South Bay, CA

Work Arrangement: Onsite

About the Role

An innovative AI infrastructure startup is seeking a Founding Engineer specializing in LLM infrastructure and inference to help build next-generation AI serving systems.

The company is developing intelligent software that analyzes AI training and inference workloads to improve performance, power efficiency, and infrastructure costs.

This is a highly hands-on engineering opportunity focused on designing, building, and optimizing distributed LLM serving infrastructure for large language models and agentic AI workloads.

The founding engineer will take significant ownership of system architecture and implementation, working directly with the founding team to develop the company's core technology. Opportunities are also available for exceptional early-career engineers with strong computer science fundamentals and demonstrated technical ability.

Key Responsibilities

• Design, build, and optimize scalable infrastructure for hosting and serving large language models across multiple machines.
• Own architecture and implementation of distributed LLM inference systems, with significant technical autonomy.
• Develop infrastructure supporting agentic AI workloads and evolving inference requirements.
• Analyze system performance across compute, memory, and distributed execution.
• Investigate bottlenecks affecting inference latency, throughput, hardware utilization, and overall efficiency.
• Optimize inference serving strategies, including batching, scheduling, caching, and parallel execution.
• Develop performance measurement and benchmarking methodologies for representative LLM workloads.
• Evaluate and improve KV cache management, memory utilization, and inference execution efficiency.
• Collaborate with performance and infrastructure engineers to translate workload-level optimizations into system-wide improvements.
• Work directly with the founding team to influence architecture, engineering decisions, and development of the serving platform.

Required Qualifications

• Hands-on experience building, implementing, or optimizing LLM inference or model-serving infrastructure.
• Strong understanding of distributed systems and multi-machine execution.
• Understanding of how model execution interacts with compute, memory, and machine-level constraints.
• Ability to identify performance bottlenecks and implement measurable improvements in latency, throughput, or hardware utilization.
• Strong software engineering fundamentals and experience debugging complex systems.
• Ability to explain architectural decisions, technical trade-offs, and measured performance improvements.
• Willingness to work across infrastructure, inference engines, runtime behavior, and system optimization in an early-stage engineering environment.
• For founding-level candidates, demonstrated ability to independently design and build substantial systems.

For Junior Candidates: Strong computer science fundamentals, substantial engineering projects, and demonstrated ability to learn complex technical concepts quickly.

Preferred Qualifications

• Experience developing or contributing to LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or llm-d.
• Understanding of inference optimization techniques, including KV cache management, continuous batching, PagedAttention, and FlashAttention.
• Experience optimizing inference latency, including time to first token (TTFT), throughput, and serving efficiency.
• Familiarity with distributed execution techniques such as tensor parallelism, pipeline parallelism, and expert parallelism.
• Experience with distributed machine learning frameworks such as DeepSpeed, Megatron, or FSDP.
• Knowledge of GPU performance profiling, memory management, and hardware utilization.
• Experience with agentic AI workloads and dynamic inference demands.
• Background in high-performance computing, distributed systems, or AI infrastructure.

Why Join?

• Take foundational ownership of LLM serving infrastructure and distributed inference systems.
• Work directly with an experienced founding team with backgrounds in advanced AI hardware and accelerator development.
• Influence the architecture and technical direction of an emerging AI infrastructure platform.
• Build infrastructure supporting LLM and agentic AI workloads.
• Solve challenging performance problems spanning distributed systems, memory management, and inference optimization.
• Join a small engineering team where individual contributions directly influence product development.
• Opportunities for both experienced founding-level engineers and exceptional junior engineers.

The team values strong engineering fundamentals, technical curiosity, independent problem-solving, and the ability to explain complex systems and performance improvements in detail.