LLM Inference Engineer — Distributed Systems

OpenTalent; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Anyscale is seeking a Distributed LLM Inference Engineer in San Francisco to push the performance boundaries of large-scale inference. You will collaborate with product teams to deliver end-to-end batch and online inference solutions, leveraging Ray Data and LLM engines.

The role requires familiarity with deep learning frameworks (PyTorch), distributed systems, and GPUs/CUDA, with opportunities to contribute to open source projects like vLLM and TensorRT-LLM.

Qualifications

• Familiarity with running ML inference at large scale with high throughput and low latency.
• Familiarity with deep learning and deep learning frameworks (e.g. PyTorch).
• Solid understanding of distributed systems, ML inference challenges.

Responsibilities

• Iterate quickly with product teams to ship end-to-end batch and online inference solutions at high scale.
• Work across the stack integrating Ray Data and LLM engine providing optimizations for low-cost large-scale ML inference.
• Integrate with open source software like vLLM and contribute improvements to open source.
• Follow state-of-the-art in open source and research, implementing best practices.

Skills

ML inference at scale

Deep learning

PyTorch familiarity

Distributed systems

Tools

Ray

vLLM

TensorRT-LLM

CUDA

Job description

Anyscale is seeking a Distributed LLM Inference Engineer in San Francisco to push the performance boundaries of large-scale inference. You will collaborate with product teams to deliver end-to-end batch and online inference solutions, leveraging Ray Data and LLM engines.

The role requires familiarity with deep learning frameworks (PyTorch), distributed systems, and GPUs/CUDA, with opportunities to contribute to open source projects like vLLM and TensorRT-LLM.