Production Engineer - Applied Machine Learning

ByteDance; San Jose (CA); On-site; USD 156,000 - 387,600; Full time

Benefits offered by this job

Medical, dental, and vision insurance

401(k) with company match

Paid parental leave

Disability coverage (short/long term)

Life insurance

Wellbeing benefits

10 paid holidays per year

10 paid sick days per year

17 days Paid Personal Time

Job summary

ByteDance is hiring a Production Engineer for Applied Machine Learning in San Jose. You will own production stability for AML training, inference, and storage systems, manage Kubernetes and GPU clusters, and drive reliability through SLOs, observability, and post-mortems.

The role emphasizes CI/CD, canary releases, capacity forecasting, and cost-aware resource management, collaborating across cross-functional teams to optimize performance.

Qualifications

• Bachelor's degree or higher in CS/SE/AI or related field.
• Proficient in Linux; Shell/Python/Go/C++ in one or more.
• Understanding ML training/inference, Kubernetes, GPUs, distributed storage.
• Experience with online troubleshooting, performance analysis, automation.
• Strong collaboration across cross-functional teams.

Responsibilities

• Ensure production stability of AML training, inference, and storage systems.
• Manage pipelines, scheduling, orchestration, and GPU clusters.
• Build and maintain SLO/SLA, observability, on-call processes, post-mortems.
• Drive CI/CD, canary releases, auto-rollback, and capacity forecasting.

Skills

Troubleshooting

Performance analysis

Automation

Linux

SLO governance

FinOps

NoSQL

Open-source infra

Cross-functional collaboration

Education

Bachelor's degree in Computer Science, Software Engineering, AI, or related fields

Tools

Shell

Python

Go

C++

Kubernetes

GPU clusters

Distributed storage

Job description

Production Engineer - Applied Machine Learning

Location: San Jose | Team: Technology | Employment Type: Regular | Job Code: A29962

Responsibilities

• System Stability & Production Management: Responsible for the production management and stability assurance of AML (Applied Machine Learning) training, inference, and storage systems. This covers core pipelines including scheduling and orchestration, K8s/GPU clusters, distributed training, online inference serving, and ParameterServer/NoSQL storage.
• Reliability Engineering: Build and maintain mechanisms for SLO/SLA, observability, alerting, on‑call processes, fault diagnosis, auto‑healing, disaster recovery, and incident reviews (post‑mortems).
• Engineering Excellence: Drive engineering capabilities such as CI/CD, canary releases, auto‑rollback, automated inspections, pre‑flight checks, capacity forecasting, and elastic auto‑scaling.
• Resource & Cost Management: Oversee resource governance across GPU/CPU/storage/network, including quota management, cost attribution, and performance tuning, to improve system availability, resource utilization, and overall R&D efficiency.

Qualifications

• Minimum Qualification(s):
• - Bachelor's degree or above in Computer Science, Software Engineering, Artificial Intelligence, or related fields.
• - Familiar with Linux and proficient in at least one of the following programming/scripting languages: Shell, Python, Go, or C++.
• - Understanding of machine learning training/inference architectures, Kubernetes, GPU clusters, or distributed storage systems.
• - Proven experience in online troubleshooting, performance analysis, and building automation platforms.
• - Strong sense of responsibility, clear logical thinking, and the ability to drive the resolution of complex issues across cross‑functional teams.
• Preferred Qualification(s):
• - Experience with large‑scale training/inference/storage platforms, SLO governance, FinOps, NoSQL, or open‑source infrastructure is highly preferred.

Job Information

The base salary range for this position in the selected city is $156,000 - $387,600 annually. Compensation may vary outside of this range depending on a number of factors, including a candidate's qualifications, skills, competencies and experience, and location. Base pay is one part of the Total Package that is provided to compensate and recognize employees for their work, and this role may be eligible for additional discretionary bonuses/incentives, and restricted stock units.

Benefits

Benefits may vary depending on the nature of employment and the country work location. Employees have day one access to medical, dental, and vision insurance, a 401(k) savings plan with company match, paid parental leave, short‑term and long‑term disability coverage, life insurance, wellbeing benefits, among others. Employees also receive 10 paid holidays per year, 10 paid sick days per year, and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure). The Company reserves the right to modify or change these benefits programs at any time, with or without notice.

Equal Opportunity / Discrimination Policy

For Los Angeles County (unincorporated) Candidates:

1. Interacting and occasionally having unsupervised contact with internal/external clients and/or colleagues;
2. Appropriately handling and managing confidential information including proprietary and trade secret information and access to information technology systems;
3. Exercising sound judgment.

Reasonable Accommodation

ByteDance is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at https://tinyurl.com/RA-request.