Remote Senior Principal LLM Architect

Recruits Lab; Washington, Baltimore (District of Columbia, MD); Remote; USD 250,000 - 350,000; Full time

Job summary

Recruits Lab is seeking a Senior Principal AI Scientist to architect, build, and pre-train foundation models from scratch. You will lead end-to-end model development across large GPU clusters, collaborating with ML researchers and systems engineers to shape the next generation of AI systems.

This role demands deep expertise in transformers, MoE, and scalable training techniques, with a focus on breakthroughs in model architecture and training stability.

Qualifications

• Proven hands-on experience pre-training large-scale LLMs or foundation models from scratch, with direct ownership of architecture and training decisions.
• Deep expertise in transformers, modern deep learning, optimization, and representation learning.
• Strong knowledge of RoPE, GQA, MoE, attention mechanisms, and scaling laws.
• Experience with distributed GPU training, including FSDP, DeepSpeed ZeRO, tensor parallelism, and pipeline parallelism.
• Hands-on experience with BF16/FP8, gradient checkpointing, training stability, and large-scale GPU infrastructure.
• A track record of solving complex AI research challenges and driving high-impact technical decisions.

Responsibilities

• Architect, develop, and pre-train large-scale LLMs and foundation models from scratch, including multimodal architectures.
• Explore transformers, attention mechanisms, mixture-of-experts (MoE), and innovative model designs.
• Address training divergence, convergence, numerical stability, and optimization across large GPU clusters.
• Advance scaling laws, training efficiency, generalization, and inference optimization.
• Partner with leading ML researchers and systems engineers while influencing architectural decisions and long-term research direction.

Job description

Recruits Lab is seeking a Senior Principal AI Scientist to architect, build, and pre-train foundation models from scratch. You will lead end-to-end model development across large GPU clusters, collaborating with ML researchers and systems engineers to shape the next generation of AI systems.

This role demands deep expertise in transformers, MoE, and scalable training techniques, with a focus on breakthroughs in model architecture and training stability.