Graduate Cloud-Native LLM Inference Engineer

ByteDance; San Jose (CA); On-site; USD 162,000 - 317,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401(k) with company match

Paid parental leave

Disability coverage

Life insurance

Wellbeing benefits

10 holidays per year

10 sick days per year

Paid Personal Time

Job summary

ByteDance is seeking engineers for its Inference Infrastructure team to advance the scale and efficiency of AI inference across multi-cloud data centers. You will work on Kubernetes-native control planes, GPU-accelerated ML platforms, and open-source tooling to enable production-grade AI workloads.

The role emphasizes collaboration across global teams, contribution to open-source, and tackling ambitious cloud-native challenges in a hyper-scale environment.

Qualifications

• PhD in Computer Science or related technical discipline.
• Strong understanding of large model inference and distributed systems.
• Hands-on experience building cloud or ML infrastructure.
• Knowledge of container/orchestration technologies (Docker, Kubernetes).
• Proficiency in at least one language (Go, Rust, Python, or C).

Responsibilities

• Design and build large-scale, container-based cluster management and orchestration systems with extreme performance, scalability, and resilience.
• Architect next-generation cloud-native GPU and AI accelerator infrastructure for cost-efficient ML platforms.
• Collaborate across teams to deliver inference solutions using vLLM, SGLang, TensorRT-LLM, and other LLM engines.
• Stay current with open source and AI/ML advances; integrate best practices into production systems.
• Write high-quality, production-ready code that is maintainable, testable, and scalable.

Skills

Large model inference

Distributed systems

Cloud infrastructure

Docker & Kubernetes

Go / Rust / Python / C

GPU acceleration

Education

PhD in Computer Science or related

Tools

vLLM

SGLang

TensorRT-LLM

Ray

Job description

ByteDance is seeking engineers for its Inference Infrastructure team to advance the scale and efficiency of AI inference across multi-cloud data centers. You will work on Kubernetes-native control planes, GPU-accelerated ML platforms, and open-source tooling to enable production-grade AI workloads.

The role emphasizes collaboration across global teams, contribution to open-source, and tackling ambitious cloud-native challenges in a hyper-scale environment.