LLM Inference Engineer

China Mobile International Limited; Kuala Lumpur; Sur place; MYR 90 000 - 130 000; Plein temps

Résumé du poste

China Mobile International Limited seeks an engineer to deploy and run LLM inference pipelines on GPU servers in Kuala Lumpur. The role focuses on setting up NVIDIA GPUs, deploying models, quantizing, and building container images to provide OpenAI-compatible APIs.

You will optimize performance, monitor bottlenecks, validate API functionality, and conduct benchmarks and stress testing to ensure high availability and low latency across services.

Qualifications

• Bachelor's degree in Computer Science or related field.
• At least one year of experience in LLM inference deployment.
• Experience with bare-metal GPU servers and Docker/Kubernetes.
• Strong knowledge of vLLM, SGLang, or TensorRT-LLM.
• Familiarity with CUDA, NCCL, RDMA/InfiniBand, KV cache, FP8, and distributed inference.
• Good Python, Linux, and shell scripting skills.
• Experience with THVV, EvalScope, or similar testing tools.
• Familiarity with OpenAI-compatible APIs, streaming, tool calls, and structured output.

Responsabilités

• Set up NVIDIA B300 GPU servers, including drivers, CUDA, NCCL, RDMA, Docker, and Kubernetes.
• Deploy vLLM, SGLang, or TensorRT-LLM and build container images.
• Convert, quantize, and shard open-source models for multi-node, multi-GPU inference.
• Provide OpenAI-compatible APIs and integrate gateways, authentication, rate limits, quotas, and caching.
• Optimize batching, KV cache, prefix cache, FP8, and distributed inference.
• Improve throughput and reduce time to first token (TTFT).
• Identify GPU, memory, network, and gateway bottlenecks.
• Optimize 128K–256K long-context workloads.
• Test API functions, model quality, performance, capacity, and stability.
• Validate streaming, tool calls, structured output, multimodal input, caching, error handling, and security.
• Run benchmarks such as AIME 2026, GPQA Diamond, MMLU-Pro, and LongBench v2.
• Analyze THVV reports, fix issues, and prepare final test reports.
• Monitor GPU usage, memory, TPM/RPM, TTFT, success rates, errors, and cache hit rates.
• Resolve OOM errors, GPU failures, rate-limit errors, timeouts, network issues, and service hangs.
• Maintain at least 99.9% monthly SLA.
• Run regular stress tests and regression tests.
• Manage customer credits, quotas, and rate limits.

Connaissances

LLM deployment

Python

Linux

Shell scripting

Formation

Bachelor's degree in Computer Science or related field

Outils

Docker

Kubernetes

CUDA

NCCL

RDMA/InfiniBand

OpenAI-compatible APIs

Description du poste

1. Deployment

• Set up NVIDIA B300 GPU servers, including drivers, CUDA, NCCL, RDMA, Docker, and Kubernetes.
• Deploy vLLM, SGLang, or TensorRT-LLM and build container images.
• Convert, quantize, and shard open-source models for multi-node, multi-GPU inference.
• Provide OpenAI-compatible APIs and integrate gateways, authentication, rate limits, quotas, and caching.

2. Performance Optimization

• Optimize batching, KV cache, prefix cache, FP8, and distributed inference.
• Improve throughput and reduce time to first token (TTFT).
• Identify GPU, memory, network, and gateway bottlenecks.
• Optimize 128K–256K long-context workloads.

3. Testing and Delivery

• Test API functions, model quality, performance, capacity, and stability.
• Validate streaming, tool calls, structured output, multimodal input, caching, error handling, and security.
• Run benchmarks such as AIME 2026, GPQA Diamond, MMLU-Pro, and LongBench v2.
• Analyze THVV reports, fix issues, and prepare final test reports.

4. Operations

• Monitor GPU usage, memory, TPM/RPM, TTFT, success rates, errors, and cache hit rates.
• Resolve OOM errors, GPU failures, rate-limit errors, timeouts, network issues, and service hangs.
• Maintain at least 99.9% monthly SLA.
• Run regular stress tests and regression tests.
• Manage customer credits, quotas, and rate limits.

Requirements

• Bachelor’s degree or above in Computer Science or a related field.
• At least one year of experience in LLM inference deployment.
• Experience with bare-metal GPU servers and Docker/Kubernetes.
• Strong knowledge of vLLM, SGLang, or TensorRT-LLM.
• Familiarity with CUDA, NCCL, RDMA/InfiniBand, KV cache, FP8, and distributed inference.
• Good Python, Linux, and shell scripting skills.
• Experience with THVV, EvalScope, or similar testing tools.
• Familiarity with OpenAI-compatible APIs, streaming, tool calls, and structured output.