Staff GenAI Inference Engineer: Optimize LLM Serving Latency

Neon; San Francisco (CA); On-site; USD 190,900 - 232,800; Full time

Benefits offered by this job

Annual performance bonus

Equity options

Comprehensive health benefits

Job summary

A leading data and AI company is seeking a Staff Software Engineer for GenAI inference to lead the architecture and optimization of the inference engine. The role requires expertise in CUDA, GPU programming, and distributed systems design. Ideal candidates will have a strong software engineering background and a proven ability to collaborate with researchers and drive architectural decisions. Competitive compensation is offered, with a salary range of $190,900 to $232,800 USD.

Qualifications

• 6+ years of experience in performance-critical systems.
• Proven track record of owning complex system components.
• Hands-on experience with CUDA and GPU programming.

Responsibilities

• Own and drive the architecture and implementation of the inference engine.
• Lead the optimization for latency and throughput across GPUs.
• Collaborate with researchers to integrate new model architectures.

Skills

CUDA programming

GPU programming

Distributed systems design

Communication skills

Performance optimization

Education

BS/MS/PhD in Computer Science

Tools

cuBLAS

cuDNN

NCCL

Job description

A leading data and AI company is seeking a Staff Software Engineer for GenAI inference to lead the architecture and optimization of the inference engine. The role requires expertise in CUDA, GPU programming, and distributed systems design. Ideal candidates will have a strong software engineering background and a proven ability to collaborate with researchers and drive architectural decisions. Competitive compensation is offered, with a salary range of $190,900 to $232,800 USD.