Lead Distributed Pre-Training Engineer for Multilingual LLMs

Barcelona Supercomputing Center; Barcelona; Presencial; EUR 85.000 - 120.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Restaurant tickets

Private health insurance

Relocation support

Descripción de la vacante

Barcelona Supercomputing Center in Barcelona is seeking a Senior Pre-training Engineer (RE3) to lead multilingual pre-training pipelines on multi-node HPC infrastructure. You will drive distributed training strategies and prototype novel architectures for large language models.

The role requires deep expertise in Megatron-LM, DeepSpeed, PyTorch FSDP or NeMo, and strong Python and English language skills. You will work in a cutting-edge HPC environment with access to MareNostrum, Leonardo, and

Formación

• Experience in pre-training large language models across multi-node, multi-GPU setups.
• Expertise with Megatron-LM, DeepSpeed, PyTorch FSDP or NeMo.
• Proficient in Python and PyTorch/DeepSpeed.
• Strong understanding of scaling laws and HPC environments.
• Excellent English (written and spoken).
• Spanish fluency is valued.

Responsabilidades

• Lead design and execution of large-scale pre-training pipelines for multilingual models on multi-node HPC infrastructure.
• Design and implement distributed training strategies (data, tensor, pipeline, and sequence parallelism) to scale training across GPUs.
• Apply scaling laws to guide compute-optimal decisions on model size and training budget.
• Research and prototype novel model architectures for pre-training at scale.
• Optimize data pipelines and tokenization for multilingual pre-training.
• Collaborate with HPC infrastructure teams to maximize cluster utilization and fault tolerance.
• Mentor junior team members on distributed pre-training practices.

Conocimientos

Large language models

Distributed training

Python

PyTorch

Deep Learning

HPC experience

English proficiency

Educación

PhD or Master’s in CS/AI

Herramientas

Megatron-LM

DeepSpeed

PyTorch FSDP

NeMo

Descripción del empleo

Barcelona Supercomputing Center in Barcelona is seeking a Senior Pre-training Engineer (RE3) to lead multilingual pre-training pipelines on multi-node HPC infrastructure. You will drive distributed training strategies and prototype novel architectures for large language models.

The role requires deep expertise in Megatron-LM, DeepSpeed, PyTorch FSDP or NeMo, and strong Python and English language skills. You will work in a cutting-edge HPC environment with access to MareNostrum, Leonardo, and