Principal ML Engineer, Training Pipelines (Hybrid)

Electronic Arts; Redwood City (CA); Hybrid; USD 180,000 - 320,000; Full time

Benefits offered by this job

Health insurance

401(k)

Paid time off

Sick time

Paid holidays

Family leave

Medical/dental/vision

Life and disability insurance

Job summary

Electronic Arts seeks a Principal ML Engineer to own and architect training input pipelines for multimodal game data, driving throughput, scalability, and reliability across accelerators. You will collaborate with researchers and central data/compute teams to translate evolving model needs into reusable training capabilities.

You will lead hands-on implementation, mentor engineers, and potentially grow into people management as the team expands, while maintaining a focus on reproducibility and

Qualifications

• 20+ years of development experience owning production distributed-training input systems with measurable throughput improvements.
• Expertise building custom PyTorch training and ingestion systems or adapting TorchTitan, Megatron Core/NeMo, or equivalent stacks for large-scale jobs.
• Expertise in FSDP2/HSDP, tensor/context/pipeline parallelism, mixed-precision training, and accelerator platforms (GPUs/TPUs/AWS Trainium).
• Pre-training experience and understanding of how model architecture, data representation, batching, and orchestration impact training-system design.
• Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators to diagnose cross-layer failures and intermittent faults.

Responsibilities

• Architect and build shared training input pipelines from game capture to accelerators, including storage layout, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery.
• Define schemas, data contracts, encodings, compression, and standards for dataset versioning, lineage, quality, coverage, and reproducible splits.
• Build deterministic loaders and checkpoint-safe iteration that recover from worker or hardware failures and data corruption with elastic restarts.
• Design reproducible dataset mixtures and balanced batches across modalities, resolutions, lengths, and capture conditions for distributed training.
• Benchmark throughput, data-wait time, step time, and accelerator utilization; analyze MFU, per-rank skew, restart time, and cost targets.

Skills

Distributed training

PyTorch training pipelines

FSDP/HSDP

Mixed-precision training

Data ingestion systems

Researcher collaboration

Tools

TorchTitan

Megatron Core/NeMo

Ray Data

NVIDIA DALI

WebDataset

Job description

Electronic Arts seeks a Principal ML Engineer to own and architect training input pipelines for multimodal game data, driving throughput, scalability, and reliability across accelerators. You will collaborate with researchers and central data/compute teams to translate evolving model needs into reusable training capabilities.

You will lead hands-on implementation, mentor engineers, and potentially grow into people management as the team expands, while maintaining a focus on reproducibility and