Senior ML Engineer | Kimchi (Inference Stack Optimization)

castaigroupinc; USA; Remote; USD 180.000 - 280.000; Vollzeit

Benefits dieser Stelle

Remote-first work

equity options

a learning budget

equipment support

extra days off

time allocated for personal developmen

Zusammenfassung

castaigroupinc is seeking a senior engineer to improve the performance and economics of large-scale language-model inference across the serving stack.

You will own from model loading to caching and distributed execution, balancing latency, throughput, and cost while guiding optimization priorities.

This role blends hands-on systems work with setting strategy, working remotely in a remote-first environment.

Qualifikationen

• Minimum five years building production machine-learning systems.
• Strong production Python skills and experience with vLLM, SGLang, or TensorRT-LLM.
• Understanding of GPU performance and quantization effects on quality, not only compression.
• Experience with multi-GPU or multi-node systems, parallelism, and collective communication.
• Ability to instrument systems and distinguish repeatable improvements from benchmark artifacts.

Aufgaben

• Tune serving engines, batching, prefill, decoding, and scheduling to raise throughput across GPU types.
• Profile time-to-first-token and token-generation latency, identify bottlenecks, and optimize compute/memory/storage/network paths.
• Improve KV-cache usage through prefix caching, paging, tiering, eviction policies, and quantization.
• Shorten model startup by improving weight-loading paths, compilation caches, and shared-filesystem reads.
• Evaluate quantization on real workloads, balancing model quality with memory and performance gains.
• Design and improve multi-GPU and multi-node inference, including placement and separation of prefill and decode; establish reproducible benchmarks and explain technical recommendations.

Kenntnisse

Production ML systems

Production Python

GPU performance profiling

Multi-GPU / multi-node

System instrumentation

Tools

vLLM

SGLang

TensorRT-LLM

Direct I/O

Shared filesystem

Jobbeschreibung

Role overview

Improve the performance and economics of large-scale language-model inference, with ownership spanning the serving stack from model loading and scheduling to caching and distributed execution. The goal is to reduce latency and cold-start time while increasing throughput and controlling cost, using measured results to guide technical decisions. The role combines hands-on systems work with responsibility for setting optimization priorities.

Responsibilities

• Tune serving engines, batching, prefill, decoding, and scheduling to raise throughput across GPU types.


• Profile time-to-first-token and token-generation latency, identify real bottlenecks, and optimize the responsible compute, memory, storage, network, or scheduling paths.


• Improve KV-cache usage through techniques such as prefix caching, paging, tiering, eviction policies, and quantization.


• Shorten model startup by improving weight-loading paths, compilation caches, and shared-filesystem reads.


• Evaluate quantization on real workloads, balancing model quality with memory and performance gains.


• Design and improve multi-GPU and multi-node inference, including placement and separation of prefill and decode; establish reproducible benchmarks and explain technical recommendations.

Requirements

• At least five years building production machine-learning systems, with substantial experience in inference or training infrastructure.


• Strong production Python skills and practical experience running vLLM, SGLang, or TensorRT-LLM.


• Understanding of GPU performance and experience measuring quantization effects on quality, not only compression.


• Experience with multi-GPU or multi-node systems, parallelism, and collective communication.


• Knowledge of memory and storage behavior, including page caches, direct I/O, and large reads over network filesystems.


• Ability to instrument systems and distinguish repeatable improvements from benchmark artifacts.

Nice to have

• Experience with LLM routing evaluations, cost-quality-latency trade-offs, calibration or abstention methods, or production A/B experiments with drift monitoring.

Benefits and work setup

• Remote-first work


• equity options


• a learning budget


• equipment support


• extra days off


• time allocated for personal development

The role does not include visa sponsorship.