Senior ML Engineer | Kimchi (Inference Stack Optimization)

Creandum

Deutschland

Remote

EUR 120,000 - 170,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity options
Remote-first global environment
Learning budget for professional and个人
Annual hackathon
Equipment budget
Extra days off
Team-building budget
Checkr background check not Sponsoring

Job summary

Cast AI, a Miami-headquartered, globally distributed team, seeks an inference systems engineer for Kimchi Inference to optimize throughput, latency, cold starts, and cost per token. Your work will be the inference stack’s optimization, tuning loaders, schedulers, and kernels, with impact on customer p99 and our P&L.

You will own the technical direction, experiment with caching, disaggregation, and tiered KV caches, and work across multi-GPU/multi-node deployments to squeeze maximum performance

Qualifications

  • You've built real ML systems for 5+ years with depth in inference or training infrastructure.
  • Python is production-grade: services, not scripts.
  • You've run vLLM, SGLang, or TensorRT-LLM for real workloads and understand performance on different GPUs.
  • You've measured quantization regressions, not just compression ratios.
  • Multi-GPU and multi-node setups are familiar, including tensor/expert parallelism and interconnect issues.
  • You know where bytes live (page caches, O_DIRECT, shared filesystems) and how fast they move.
  • You instrument before you optimize and can distinguish real wins from benchmarks.
  • A wide mandate excites you more than it unsettles you.

Responsibilities

  • Push throughput with continuous batching, prefill, and scheduler tuning across the engine fleet.
  • Cut latency by profiling bottlenecks and fixing compute, memory, scheduling, storage, or network limits.
  • Get more out of the KV cache with hierarchical/paged attention and eviction policies.
  • Kill cold starts with weight loaders, JIT/compile caches, and shared-filesystem paths.
  • Quantize without regressing quality (INT8/INT4, FP8/FP4 across weights/activations/KV).
  • Scale across nodes with distributed inference topologies and disaggregation strategies.
  • Set the technical direction, benchmark choices, and reproducible experiments.

Skills

ML infrastructure
Inference systems
Python production-grade
Observability
Benchmarking
TensorRT-LLM
vLLM
SGLang

Tools

vLLM
SGLang
TensorRT-LLM
Python

Job description

Why Kimchi?

Great models are raw ingredients. Kimchi is what happens when you ferment them with good engineering. We started inside CAST AI helping companies run LLMs on their own Kubernetes clusters; today we run our own coding harness on any model you pick, and we run the models underneath it ourselves.Kimchi Inference

One API line and a customer’s existing client just works: DeepSeek, GLM, Kimi, MiniMax, Nemotron served serverless from our own GPU fleet (OpenAI- and Anthropic-compatible, pay per token, spend visible per user, team, and key). When control matters, the same models run on the customer’s own cluster instead. Behind that one line sits the machinery you’ll own: SGLang and vLLM engines, B300 nodes, a shared cluster filesystem that holds every checkpoint exactly once, KV caches riding coding traffic with heavy prefix reuse, and the benchmark harness that gates every change we ship.

Made by Cast AI - a Miami-headquartered, globally distributed team across 34 countries that hit unicorn status in January 2026 with a $50B+ backed investment pushing its valuation past $1 billion.

About the role

Throughput. Latency. Cold starts. Cost per token.

Move those four numbers in the right direction and two things happen: customers get faster, cheaper inference, and our margins improve. That’s the entire thesis of this role. Every loader you tune, every scheduler decision you land, every kernel you stare at shows up in a customer’s p99 and on our P&L. Your mandate is the inference stack’s optimization, whatever it takes, and the technical direction of how we optimize it is yours to argue with data.

The problem: the checkpoints run 0.3-1.5 TB, and the "right" serving configuration depends on traffic shape, sequence-length distribution, batch dynamics, GPU SKU, memory bandwidth, and quantization tolerance, all variables that shift week to week. Most teams pick a configuration once, over-provision GPUs, and absorb the cost. We don’t: we took Kimi-K3 (1.5 TB) from over an hour cold-to-serving to under twelve minutes with a shared filesystem, O_DIRECT weight loaders, and node-local JIT caches, and we’re nowhere near done. Prefill/decode disaggregation and tiered KV caches are next.

Stack: Python; SGLang; vLLM; TensorRT-LLM; PyTorch; fastsafetensors; CUDA-adjacent tooling; Kubernetes; virtiofs shared filesystems; ClickHouse; OpenTelemetry; Prometheus; Grafana; GitLab CI.

Requirements
  • You’ve built real ML systems for 5+ years, and your portfolio shows depth in inference or training infrastructure, not just model training notebooks.
  • Your Python is production-grade: services, not scripts.
  • You’ve run vLLM, SGLang, or TensorRT-LLM for real, and you have a working mental model of why an engine performs the way it does on a given GPU.
  • You’ve measured quantization regressions, not just compression ratios.
  • Multi-GPU and multi-node setups hold no fear: tensor and expert parallelism, collective communication, and their practical failure modes.
  • You know where the bytes live (page caches, O_DIRECT, network filesystems under terabyte-scale reads) and how fast they move.
  • You instrument before you optimize, and you can tell a real win from a benchmark artifact.
  • A wide mandate excites you more than it unsettles you.
Nice to haves
  • You’ve worked on LLM routing specifically: RouterBench-class evals, cost/quality/latency Pareto fronts.
  • Calibration, abstention, or conformal methods in production decision systems.
  • You’ve run online A/B loops with drift monitoring on a model that touches real money.
Responsibilities
  • Push throughput. Continuous batching, chunked prefill, speculative decoding with draft models, CUDA-graph capture, and scheduler tuning across the engine
    fleet. Find the ceiling on each GPU SKU, then raise it.
  • Cut latency. Attack TTFT and inter-token latency separately; profile to find the actual bottleneck (compute, memory bandwidth, scheduling, storage, or
    network) and fix it, not the one you assumed.
  • Get more out of the KV cache. Hierarchical and paged attention, prefix caching on heavy-reuse coding traffic, eviction policies, quantized KV. A lot of the
    unrealized throughput lives here, and you’ll own it.
  • Kill cold starts. Weight loaders (O_DIRECT, fastsafetensors-class), JIT and compile caches, shared-filesystem read paths. A 1.5 TB checkpoint has no business
    taking an hour to reach serving.
  • Quantize without regressing quality. INT8/INT4, FP8/FP4 across weights, activations, and KV, measured on real workloads, not just perplexity benchmarks.
  • Scale across nodes. Distributed inference topologies, network-aware placement, prefill/decode disaggregation, checkpointing strategies that don’t bottleneck
    on storage or interconnect.
  • Set the technical direction. Decide what we benchmark, what we adopt, and what we build ourselves. Bring the team along with strong writeups and reproducible
    experiments.
What’s in it for you?
  • Competitive salary (depending on the level of experience).
  • Enjoy a flexible, remote-first global environment.
  • Collaborate with a global team of cloud experts and innovators, passionate about pushing the boundaries of Kubernetes technology
  • Equity options.
  • Get quick feedback with a fast-paced workflow. Most feature projects are completed in 1 to 4 weeks.
  • Spend 10% of your work time on personal projects or self-improvement.
  • Learning budget for professional and personal development - including access to international conferences and courses that elevate your skills.
  • Annual hackathon to spark new ideas and strengthen team bonds.
  • Team-building budget and company events to connect with your colleagues.
  • Equipment budget to ensure you have everything you need.
  • Extra days off to help maintain a healthy work-life balance.
Hiring process
  • Screening call with Recruiter
  • Hiring Manager interview
  • Technical interview (system design)
  • Live coding
  • Culture Check interview with an executive

As part of our standard hiring process, we would like to inform you that a background check may be conducted at the final stage of recruitment through our third-party provider, Checkr.
Please note that Cast AI does not provide any form of visa sponsorship/work permit.

#LI-Remote

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Product Manager | Kimchi AI Coding Platform| Europe
Product Manager | Kimchi AI Coding Platform| Europe

Kimchi • Germany

Remote
EUR 107,000 - 161,000
Competitive salary
Remote-first global environment
Equity options
+2
Senior Applied ML Engineer | Kimchi (Model Router)
Senior Applied ML Engineer | Kimchi (Model Router)

Creandum • Germany

Remote
EUR 110,000 - 170,000
Equity options
Flexible remote-first environment
Learning budget for development
+3
Senior Applied ML Engineer | Kimchi (Model Router)
Senior Applied ML Engineer | Kimchi (Model Router)

Kimchi • Germany

Remote
EUR 90,000 - 150,000
Equity options
Learning budget
Flexible remote work
+1
Senior Software Engineer | Kimchi (API Platform Team)
Senior Software Engineer | Kimchi (API Platform Team)

Creandum • Germany

Remote
EUR 73,000 - 100,000
Competitive salary
Remote-first environment
Equity options
+5
Product Manager | Kimchi AI Coding Platform| Europe
Product Manager | Kimchi AI Coding Platform| Europe

Cast AI • Germany

Remote
EUR 110,000 - 165,000
Competitive salary
Equity options
Learning budget for development
+3
Senior Software Engineer | AI Agents & Cloud Platform (Cast Pilot team)
Senior Software Engineer | AI Agents & Cloud Platform (Cast Pilot team)

Cast • Germany

Remote
EUR 73,000 - 100,000
Competitive salary
Equity options
Remote-first global environment
+5
Sales Engineer - Germany
Sales Engineer - Germany

castaigroupinc • Germany

Remote
EUR 90,000 - 130,000
Remote-first environment
Equity options
Premium health insurance
+4
Infrastructure/Systems Engineer
Infrastructure/Systems Engineer

orcristtechnologies • Berlin

Remote
EUR 90,000 - 140,000
Remote-first, Germany-wide
Flexible hours
Your setup, your way
+6
Senior ML Engineer - Real-Time Inference & Systems
Senior ML Engineer - Real-Time Inference & Systems

Inworld AI • Germany

On-site
USD 120,000 - 180,000
AI Infrastructure Engineer
AI Infrastructure Engineer

Vcluster • Germany

Remote
EUR 90,000 - 130,000
Competitive Salary
Premium Insurance
Flexible Working Schedule
+1