Cloud Inference Engineer

SupportFinity™; San Francisco (CA); On-site; Full time

Job summary

A pioneering AI technology firm in San Francisco is seeking a founding member to optimize and serve models on Luminal Cloud. The role involves deploying models with advanced optimization techniques, conducting performance reviews, and enhancing scheduling processes. Ideal candidates are experienced in CUDA and GPU optimization, with hands-on knowledge of vLLM, SGLang, or TensorRT-LLM. A degree is not required, reflecting a modern approach to tech recruitment.

Qualifications

• Proficiency in CUDA and GPU optimization techniques.
• Experience with vLLM, SGLang, or TensorRT-LLM is preferred.
• Understanding of KV caching and distributed compute is a plus.

Responsibilities

• Deploy and tune models with optimizations like KV caching and batch processing.
• Conduct model performance reviews to assess efficiency.
• Improve processes for scheduling and autoscaling.

Skills

CUDA + GPU inference optimization

vLLM, SGLang, or TensorRT-LLM experience

KV caching

distributed compute

no degree required

Job description

Qualifications

• CUDA + GPU inference optimization
• vLLM, SGLang, or TensorRT-LLM experience
• KV caching, paged attention, batching, token streaming, etc.
• Distributed compute (with GPUs is a super plus)
• No degree required

Company

Luminal (YC S25) builds an AI compiler and serving stack that makes models 10x faster and production ready with one line.

Role

Founding, on site in downtown SF. Ship low latency, high throughput model serving on Luminal Cloud.

Day To Day Responsibilities

• Deploy and tune models with optimizations like KV caching, paged attention, sequence packing, etc.
• Conducting model performance reviews
• Improve scheduler, batcher, autoscaling; profile latency, cost, utilization
• Sometimes write kernels