Engineering Manager, ML Training Infrastructure

Woven by Toyota; Palo Alto (CA); On-site; USD 140,000 - 260,000; Full time

Benefits offered by this job

Health, dental and vision coverage

401(k) program

Flexible vacation policy

Family planning and care benefits

Job summary

Woven by Toyota, based in California, seeks an Engineering Manager to lead the ML Training Infrastructure team. You will shape technical direction, foster an engineering culture, and drive the long-term evolution of our ML training platform for scalable, reliable model development.

You will mentor a team of 7–8 engineers, collaborate with cross‑functional partners, and own the architecture of distributed GPU clusters, Kubernetes, and ML pipelines to accelerate innovation.

Qualifications

• 8+ years of software engineering experience
• At least 2 years in a leadership role
• Mentoring high‑performing software or platform teams
• Architecting, deploying, and maintaining large‑scale infrastructure
• Deep Kubernetes expertise
• Hands‑on exposure to cloud platform architectures and distributed systems
• Background in ML model development platforms or data pipelines
• Practical understanding of GPU cluster management and scheduling
• Operational background with high availability, incident reviews, and monitoring enhancements
• Excellent interpersonal abilities to align roadmaps across partners and leadership

Responsibilities

• Lead engineers and foster a collaborative and innovative environment.
• Manage and nurture a group of around 7–8 engineers, providing technical vision and growth.
• Establish the technical vision and roadmap for ML training infrastructure.
• Direct the architecture, construction, and operation of resilient systems for large ML workloads.
• Construct and maintain Kubernetes-driven compute clusters and GPU fleets.
• Oversee and optimize GPU resource scheduling, utilization, and cost performance.
• Advance distributed model training capabilities using Ray and related tools.
• Enhance workflow orchestration with Airflow, Temporal, and similar pipelines.
• Elevate system resilience, monitoring, and operational productivity of the ML stack.
• Collaborate with AD/ADAS, Woven City, and platform partners to ship infrastructure solutions.
• Translate product needs into executable engineering milestones with stakeholders.
• Institute rigorous software practices: reviews, testing, CD, and live site management.
• Engage in production support, post-incident reviews, and continuous refinements.
• Balance short-term feature requests with long-term architectural stability.
• Foster an inclusive, excellence-driven engineering culture.

Skills

Leadership

Software engineering

Kubernetes

Cloud platforms

Distributed systems

GPU infrastructure

Mentoring

Golang

English/Japanese bilingual

Tools

Kubernetes

Ray

Airflow

Temporal

Job description

Woven by Toyota is enabling Toyota’s once-in-a-century transformation into a mobility company. Inspired by a legacy of innovating for the benefit of others, our mission is to challenge the current state of mobility through human‑centric innovation — expanding what “mobility” means and how it serves society. Our work centers on four pillars: AD/ADAS, our autonomous driving and advanced driver assist technologies; Arene, our software development platform for software‑defined vehicles; Woven City, a test course for mobility; and Cloud & AI, the digital infrastructure powering our collaborative foundation. Business‑critical functions empower these teams to execute, and together, we’re working toward one bold goal: a world with zero accidents and enhanced well‑being for all.

About The Team

Enterprise AI is a platform that provides end‑to‑end machine learning tooling experience to support and accelerate machine learning development, including autonomous driving and other related projects. Our platform serves customers as a standardized machine learning platform within Woven by Toyota as the larger Toyota Group companies.

The Enterprise AI ML Training Infrastructure team builds and operates the infrastructure that enables engineers and researchers to train, evaluate, and iterate on machine learning models at scale. Our customers include teams working on AD/ADAS and Woven City, where large‑scale machine learning workloads, simulation, and data processing are critical to developing the next generation of mobility technologies.

The team operates at the intersection of machine learning, distributed systems, cloud infrastructure, and developer platforms. Our systems include large‑scale GPU compute environments, Kubernetes‑based cluster infrastructure, and workflow and pipeline orchestration technologies such as Ray, Airflow, and Temporal.

We are looking for an Engineering Manager to lead a team to help shape the technical direction, engineering culture, and long‑term evolution of our ML training infrastructure platform.

WHO ARE WE LOOKING FOR?

As the Engineering Manager, ML Training Infrastructure, you will lead the team responsible for building and operating the infrastructure that powers machine learning training workloads across Woven by Toyota.

You will combine engineering leadership with strong technical judgment. You will work closely with engineering, research, and product stakeholders to understand customer needs, translate them into scalable technical solutions, and ensure that our platform is reliable, efficient, secure, and easy to use.

This role is particularly suited to an engineering leader who enjoys working on complex infrastructure problems and is comfortable operating across Kubernetes, distributed systems, GPU infrastructure, cloud platforms, and machine learning workloads.

You will also play an important role in building a strong engineering culture, developing engineers, establishing effective engineering practices, and partnering with our customers to continuously improve the platform.

What You Will Enable

In this role, you will help build the infrastructure that enables teams across Woven by Toyota to train and operate increasingly large and sophisticated machine learning workloads.

Your work will directly support engineers and researchers working on AD/ADAS, Woven City, and other mobility initiatives by making compute resources easier to access, workloads more reliable, and ML development faster and more scalable.

You will have the opportunity to shape both the technical architecture of a critical ML infrastructure platform and the engineering culture of the team building it.

Responsibilities

• Lead engineers and foster a collaborative and innovative environment.
• Manage and nurture a group of around 7–8 engineers, offering technical vision, mentorship, professional growth, and effective project delivery.
• Establish and articulate the technical vision and future roadmap for machine learning training infrastructure.
• Direct the architecture, construction, and operation of resilient systems supporting massive ML model training workloads.
• Construct and maintain Kubernetes‑driven compute clusters and container ecosystems tailored for machine learning workloads.
• Oversee and optimize GPU fleets, addressing cluster scheduling, resource utilization, system stability, and cost performance.
• Advance distributed model training capabilities and workload management leveraging frameworks like Ray and adjacent tools.
• Enhance workflow orchestration platforms by deploying and supporting solutions such as Airflow, Temporal, and similar pipeline frameworks.
• Elevate the system resilience, monitoring insights, elasticity, and operational productivity of our ML training stack.
• Work closely alongside AD/ADAS, Woven City, and internal platform partners to gather feature requests and ship infrastructure solutions that accelerate development.
• Partner with cross‑functional engineering groups to specify platform interfaces, APIs, system architectures, and operational standards.
• Translate high‑level organizational objectives and user requirements into concrete, executable engineering milestones with product leads.
• Institute rigorous software practices across implementation, code reviews, automated testing, continuous deployment, and live site management.
• Engage in production support workflows, lead post‑incident reviews, and champion system refinements based on operational findings.
• Balance immediate client feature requests against long‑term architectural stability, technical hygiene, and system maintainability.
• Foster an inclusive, highly collaborative, and excellence‑driven engineering culture throughout Woven by Toyota.

Minimum Qualifications

• 8+ years of experience in software engineering, with at least 2 years in a leadership role.
• Proven track record mentoring and managing high‑performing software or platform teams.
• Demonstrated expertise in architecting, deploying, and maintaining large‑scale mission‑critical infrastructure.
• Deep technical mastery of Kubernetes orchestration alongside containerized runtime systems.
• Hands‑on exposure to cloud platform architectures and highly resilient distributed systems.
• Background engineering platform foundation layers designed for machine learning model development or data pipeline intensive workloads.
• Practical understanding of massive GPU cluster management, hardware scheduling algorithms, and capacity efficiency tuning.
• Operational background overseeing high‑availability environments, leading post‑incident reviews, and advancing monitoring capabilities.
• Excellent interpersonal abilities to align technical roadmaps across diverse engineering partners and leadership stakeholders.

NICE TO HAVES

• Experience working with Japanese customers or clients.
• Experience with Golang development (especially for cloud services)
• Experience in the automotive industry or knowledge of automotive software development (e.g., ASPICE, V-model).
• Experience working in environments with a focus on security and safety.
• English/Japanese bilingual ability

The base pay for this position ranges from $ 140,000 - $ 260,000 a year.

Your base salary is one part of your total compensation. We offer a base salary, short term and long term incentives, and a comprehensive benefits package. The total compensation offered to an employee will be dependent upon the individual's skills, experience, qualifications, location, and level.

What We Offer

• Excellent health, wellness, dental and vision coverage
• A rewarding 401k program
• Flexible vacation policy
• Family planning and care benefits

Our Commitment

• We are an equal opportunity employer and value diversity.

Any information we receive from you will be used only in the hiring and onboarding process. Please see our privacy notice for more details.