LLM Inference Frameworks & Optimizations Engineer

Together AI; San Francisco (CA); On-site; USD 160,000 - 230,000; Full time

Benefits offered by this job

Equity

Health insurance

Benefits package

Job summary

Together AI is building state-of-the-art LLM inference infrastructure to enable scalable, low-latency deployment for multimodal models. You will design and optimize distributed inference engines, collaborate with hardware and research teams, and push performance, scalability and cost-efficiency.

We seek an engineer with 3+ years in deep learning inference or HPC, proficient in Python and C++/CUDA, and hands-on experience with TensorRT, MoE and GPU optimization.

Qualifications

• Must have 3+ years in deep learning inference frameworks or high-performance computing.
• Proficient in Python and C++/CUDA for high-performance inference.
• Familiar with LLM inference frameworks (e.g., TensorRT‑LLM, vLLM, SGLang, TGI).
• Background in GPU programming, compilers, quantization or GPU cluster scheduling.

Responsibilities

• Design and develop fault‑tolerant, high‑concurrency distributed inference engine for text, image, and multimodal models.
• Implement and optimize distributed inference strategies, including MoE parallelism, tensor parallelism, and pipeline parallelism.
• Apply CUDA graph optimizations, TensorRT/TRT‑LLM graph optimizations, and PyTorch‑based compilation to enhance efficiency and scalability.

Skills

Python

C++/CUDA

DL inference

TensorRT

Mooncake KV

GPU/TPU

Tools

Kubernetes

Job description

Together AI is building state-of-the-art LLM inference infrastructure to enable scalable, low-latency deployment for multimodal models. You will design and optimize distributed inference engines, collaborate with hardware and research teams, and push performance, scalability and cost-efficiency.

We seek an engineer with 3+ years in deep learning inference or HPC, proficient in Python and C++/CUDA, and hands-on experience with TensorRT, MoE and GPU optimization.