ML Systems Architect: Multimodal Training Pipelines

Electronic Arts Ltd; Redwood City (CA); On-site; USD 227,000 - 345,000; Full time

Benefits offered by this job

Paid time off

Sick time

Holidays

Family leave

Medical/dental/vision insurance

Life insurance

Disability insurance

401(k)

Job summary

Electronic Arts seeks a Principal ML Engineer, Training Pipelines, to own and shape the data paths that feed large-scale multimodal model training across diverse game data. You will build robust input pipelines, schemas, and deterministic loaders while leading a team and collaborating with researchers to translate evolving requirements into scalable infrastructure.

You will design and optimize batch composition, data quality, and reproducibility, while benchmarking throughput, latency, and

Qualifications

• 20+ years of development experience in distributed-training input systems with measurable improvements.
• Expertise building custom PyTorch training/ingestion systems or adapting TorchTitan, JAX/XLA with MaxText, Megatron Core/NeMo.
• Expertise in FSDP/HSDP, tensor/context/pipeline parallelism, mixed-precision training on GPU/TPU/AWS Trainium.
• Pre-training experience and understanding of model architecture, data representation, batching, and orchestration impact on efficiency.
• Expertise in large-scale multimodal streaming or hardware-accelerated ingestion with Ray Data, Mosaic Streaming, WebDataset, NVIDIA DALI, NVDEC.
• Deep knowledge of sharding, sampling, packing, caching, prefetching, deterministic data mixing, and reproducible training/evaluation.
• Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators, including cross-layer failures

Responsibilities

• Architect and build shared training input pipelines from game capture to accelerators, with storage, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery across frameworks.
• Define data schemas, contracts, encodings, compression, and standards for dataset versioning, lineage, quality, and reproducible splits.
• Build deterministic loaders and checkpoint-safe iteration that recover from failures and support elastic restarts and changes in cluster size.
• Design reproducible dataset mixtures and balanced batches across modalities, resolutions, sequence lengths, and capture conditions.
• Benchmark throughput and accelerator utilization; model FLOPs utilization and scaling efficiency across thousands of accelerators.
• Diagnose bottlenecks across storage, network I/O, CPU, memory, decoding, preprocessing, and accelerators; handle stragglers and faulty hardware.
• Lead training-pipeline architecture reviews, mentor engineers, and align priorities with research leads; translate requirements into scalable shared infrastructure.

Skills

Distributed training

PyTorch

JAX/XLA

Megatron Core/NeMo

FSDP/HSDP

Tensor/pipeline parallelism

Mixed-precision training

Data ingestion systems

Tools

Ray Data

Mosaic Streaming

WebDataset

NVIDIA DALI

NVDEC

Job description

Electronic Arts seeks a Principal ML Engineer, Training Pipelines, to own and shape the data paths that feed large-scale multimodal model training across diverse game data. You will build robust input pipelines, schemas, and deterministic loaders while leading a team and collaborating with researchers to translate evolving requirements into scalable infrastructure.

You will design and optimize batch composition, data quality, and reproducibility, while benchmarking throughput, latency, and