Tech Lead - AI Inference

wekatest; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Life insurance

Retirement benefits

Flexible time off

Sick leave

Statutory leave

Job summary

wekatest is seeking a lead software engineer to head a small team delivering high-throughput, low-latency inference infrastructure for large language models. You will design, build, and optimize systems across NVMe storage, GPU memory, and network paths, while mentoring three engineers and driving production reliability.

The role combines hands-on development with technical direction, requiring strong Python/C++ skills, CUDA expertise, and experience deploying GPU workloads on Kubernetes in a

Qualifications

• 5+ years of professional software engineering experience including leading engineers and owning complex production systems.
• Hands-on knowledge of LLM serving, including KV cache reuse, continuous batching, disaggregated prefill and decode, and tiered GPU memory.
• Strong Python and C++ skills and a solid understanding of CUDA, GPU memory management, and high-performance I/O.

Responsibilities

• Own technical decisions and delivery for inference infrastructure, including NVMe-backed token storage, GPU Direct Storage data paths, and model serving components.
• Guide three developers through system design, implementation, and production delivery while maintaining standards for reliability and code quality.
• Contribute directly to production systems using Python, C++, CUDA, and the surrounding infrastructure stack.
• Address challenges such as separating prompt processing from token generation, persistent KV caching beyond GPU memory, RDMA transport, and tiered GPU memory.
• Coach engineers through regular one-to-ones, career discussions, and sprint reviews.
• Evaluate emerging inference frameworks and tools through benchmarking, helping determine when to adopt, build, or change direction.

Skills

Python

C++

CUDA

Kubernetes

Mentoring/leadership

Tools

GPUDirect Storage

RDMA

NVMe data paths

GPU storage

Bare-metal GPU clusters

Job description

Role overview

Lead a small engineering team building high-throughput, low-latency infrastructure for serving large language models. The role combines hands-on systems development with technical direction, delivery ownership, and people management, working across serving frameworks, GPU memory, storage, and networking.

Responsibilities

• Own technical decisions and delivery for inference infrastructure, including NVMe-backed token storage, GPU Direct Storage data paths, and model serving components.
• Guide three developers through system design, implementation, and production delivery while maintaining standards for reliability and code quality.
• Contribute directly to production systems using Python, C++, CUDA, and the surrounding infrastructure stack.
• Address challenges such as separating prompt processing from token generation, persistent KV caching beyond GPU memory, RDMA transport, and tiered GPU memory.
• Coach engineers through regular one-to-ones, career discussions, and sprint reviews.
• Evaluate emerging inference frameworks and tools through benchmarking, helping determine when to adopt, build, or change direction.

Requirements

• At least five years of professional software engineering experience, including leading engineers and owning complex production systems.
• Hands-on knowledge of LLM serving, including KV cache reuse, continuous batching, disaggregated prefill and decode, and tiered GPU memory.
• Strong Python and C++ skills and a solid understanding of CUDA, GPU memory management, and high-performance I/O.
• Experience with GPU storage and transport technologies such as GPUDirect Storage, RDMA, and NVMe data paths.
• Experience deploying and scaling GPU workloads on Kubernetes, with familiarity in bare-metal GPU clusters and distributed storage.
• Demonstrated ability to mentor engineers and balance technical delivery with team development.
• Track record of building reliable, high-throughput systems and improving engineering practices.

Nice to have

• Experience in AI/ML infrastructure or high-performance computing.
• Familiarity with serving stacks such as vLLM, LMCache, NIXL, or comparable frameworks; Rust experience is also useful.
• Experience with SGLang or TensorRT-LLM and GPU clusters using H100- or A100-class accelerators.

Benefits and work setup

• The source mentions potential medical, dental, vision, and life insurance, retirement benefits, flexible time off, sick leave, and applicable statutory leave.

The role involves hands-on work with a three-person engineering squad.