Lead ML Engineer – Scalable GPU Training on Cloud

JP Morgan Chase; Palo Alto (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

JPMorganChase in Palo Alto seeks a Lead Software Engineer to design, build, and operate scalable ML training systems on AWS and cloud platforms. You will drive end-to-end training pipelines, run GPU workloads, and ensure governance and observability across environments.

You will collaborate with data engineering and platform teams to standardize interfaces, improve developer experience, and advance AI-assisted engineering practices while maintaining strong security and compliance.

Qualifications

• 5+ years of software engineering experience with ML systems.
• Experience running ML training in cloud environments and debugging across infra.
• Strong Python and engineering practices (testing, reviews, modular design).
• Experience building automation/CI for ML codebases and deployment workflows.
• Hands-on with deep learning training workflows (PyTorch or TensorFlow).
• Understanding of training performance, stability, data loading, and reproducibility.
• Experience with distributed training concepts (DDP/FSDP/DeepSpeed).
• Ability to profile/optimize training systems (CPU/GPU/memory/I/O).
• Experience with Kubernetes and AWS (EKS/ECR, S3, IAM, VPC, CloudWatch).
• Demonstrated use of AI-assisted software development tools with governance.

Responsibilities

• Design, build, and maintain end-to-end ML training platform.
• Run and optimize GPU training workloads and improve throughput.
• Build and operate training infra on Kubernetes (EKS and others).
• Enable Gen AI/LLM training and fine-tuning workflows with governance.
• Implement observability for training systems: metrics, logs, dashboards, alerting.
• Partner with data engineering and platform teams to define interfaces and guardrails.
• Improve developer experience with containers, CI/CD, templates, docs, self-service workflows.
• Promote enterprise-authorized AI-assisted engineering practices with validation standards.
• Apply SDLC tooling to improve automation and value.

Skills

Python

ML Training in Cloud

Kubernetes

AWS

GPU Training

CI/CD for ML

Observability

AI Governance

Tools

Kubernetes

AWS

PyTorch

TensorFlow

Docker

CI/CD tooling

S3

EKS

CloudWatch

Job description

JPMorganChase in Palo Alto seeks a Lead Software Engineer to design, build, and operate scalable ML training systems on AWS and cloud platforms. You will drive end-to-end training pipelines, run GPU workloads, and ensure governance and observability across environments.

You will collaborate with data engineering and platform teams to standardize interfaces, improve developer experience, and advance AI-assisted engineering practices while maintaining strong security and compliance.