ML Infra Production Engineer: Training, Inference & Kubernetes

BYTEDANCE PTE. LTD.; Singapore; On-site; SGD 180,000 - 260,000; Full time

Job summary

ByteDance PTE. LTD. in Singapore is seeking a Data-AML engineer to lead production operations for AML training, inference, and storage systems, including scheduling, orchestration, and distributed training. You will work across Kubernetes/GPU clusters and NoSQL storage, ensuring high availability and automated recovery.

You will drive CI/CD improvements, capacity forecasting, and elastic scaling while governing resources across GPU/CPU/storage. Strong ownership and collaboration are essential.

Qualifications

• Bachelor's degree or above in Computer Science, Software Engineering, Artificial Intelligence, or a related field.
• Proficient in Linux and skilled in Shell, Python, Go, or C++.
• Familiar with machine learning training and inference architectures, Kubernetes, GPU clusters, or distributed storage systems.
• Experienced in production issue troubleshooting, performance analysis, and automation platform development.
• Strong sense of ownership, solid analytical and problem-solving skills, and the ability to drive cross-functional collaboration to resolve complex technical challenges.

Responsibilities

• Responsible for production operations management and stability assurance of AML training, inference, and storage systems, covering core pipelines such as scheduling and orchestration, Kubernetes (K8s)/GPU clusters, distributed training, online inference serving, and Parameter Server/NoSQL storage.
• Build and maintain SLO/SLA frameworks, observability, alerting, on-call processes, incident diagnosis, self-healing mechanisms, disaster recovery, and post-incident review (postmortem) practices.
• Drive engineering capabilities including CI/CD, canary/gradual deployments, automated rollback, system health inspections, pre-flight checks, capacity forecasting, and elastic scaling.
• Lead resource governance and optimization across GPU, CPU, storage, and network infrastructure, including quota management, cost attribution, and performance tuning, to improve system availability, resource utilization, and engineering productivity.

Skills

Linux

Shell

Python

Go

C++

ML training/inference

Kubernetes

Distributed storage

Issue troubleshooting

Automation platform development

Ownership & collaboration

Education

Bachelor's degree in Computer Science / Software Engineering / AI

Tools

Kubernetes ecosystem

Mooncake KV Caches

Job description

ByteDance PTE. LTD. in Singapore is seeking a Data-AML engineer to lead production operations for AML training, inference, and storage systems, including scheduling, orchestration, and distributed training. You will work across Kubernetes/GPU clusters and NoSQL storage, ensuring high availability and automated recovery.

You will drive CI/CD improvements, capacity forecasting, and elastic scaling while governing resources across GPU/CPU/storage. Strong ownership and collaboration are essential.