Senior Principal AI Scientist

Recruits Lab; Washington, Baltimore (District of Columbia, MD); Remote; USD 250,000 - 350,000; Full time

Job summary

Recruits Lab is seeking a Senior Principal AI Scientist to architect, build, and pre-train foundation models from scratch. You will lead end-to-end model development across large GPU clusters, collaborating with ML researchers and systems engineers to shape the next generation of AI systems.

This role demands deep expertise in transformers, MoE, and scalable training techniques, with a focus on breakthroughs in model architecture and training stability.

Qualifications

• Proven hands-on experience pre-training large-scale LLMs or foundation models from scratch, with direct ownership of architecture and training decisions.
• Deep expertise in transformers, modern deep learning, optimization, and representation learning.
• Strong knowledge of RoPE, GQA, MoE, attention mechanisms, and scaling laws.
• Experience with distributed GPU training, including FSDP, DeepSpeed ZeRO, tensor parallelism, and pipeline parallelism.
• Hands-on experience with BF16/FP8, gradient checkpointing, training stability, and large-scale GPU infrastructure.
• A track record of solving complex AI research challenges and driving high-impact technical decisions.

Responsibilities

• Architect, develop, and pre-train large-scale LLMs and foundation models from scratch, including multimodal architectures.
• Explore transformers, attention mechanisms, mixture-of-experts (MoE), and innovative model designs.
• Address training divergence, convergence, numerical stability, and optimization across large GPU clusters.
• Advance scaling laws, training efficiency, generalization, and inference optimization.
• Partner with leading ML researchers and systems engineers while influencing architectural decisions and long-term research direction.

Job description

Build the Models. Push the Boundaries. Redefine What's Possible in AI.

Location: Remote | United States

The Opportunity

We're partnering with an industry-leading AI technology company advancing the next generation of foundation models, generative AI, and intelligent systems deployed at global scale.

We're searching for an exceptional Senior Principal AI Scientist who has personally architected, built, and pre-trained large language models (LLMs) or foundation models from scratch.

This isn't about fine-tuning existing models or building applications around someone else's AI. It's about developing the foundational intelligence itself.

You’ll tackle some of the hardest problems in modern AI, pioneer new approaches to model training, and have a direct impact on the architecture and capabilities of next-generation AI systems.

What You’ll Do

• Build From Scratch: Architect, develop, and pre-train large-scale LLMs and foundation models, including emerging multimodal architectures.
• Advance Model Architectures: Explore transformers, attention mechanisms, mixture-of-experts (\"MoE\"), and innovative model designs.
• Solve Frontier-Scale Challenges: Address training divergence, convergence, numerical stability, and optimization across large GPU clusters.
• Push Model Performance: Advance scaling laws, training efficiency, generalization, and inference optimization.
• Drive Technical Innovation: Partner with leading ML researchers and systems engineers while influencing critical architectural decisions and long-term research direction.

What You’ll Bring

• Essential: Proven hands-on experience pre-training large-scale LLMs or foundation models from scratch, with direct ownership of architecture and training decisions.
• Deep expertise in transformers, modern deep learning, optimization, and representation learning.
• Strong knowledge of RoPE, GQA, \"MoE\", attention mechanisms, and scaling laws.
• Experience with distributed GPU training, including FSDP, DeepSpeed ZeRO, tensor parallelism, and pipeline parallelism.
• Hands-on experience with BF16/FP8, gradient checkpointing, training stability, and large-scale GPU infrastructure.
• A track record of solving complex AI research challenges and driving high-impact technical decisions.

Highly Valued: Experience with multimodal foundation models, RLHF, DPO, GRPO, and significant contributions to foundation model research.

Why This Opportunity?

Most AI roles focus on using existing models. This one is about creating what comes next.

You’ll have the opportunity to solve ambitious research challenges, influence foundational model architecture, and help advance AI capabilities powering real-world systems worldwide.

If you’ve personally built and trained foundation models from scratch and want to push the limits of modern AI research, we’d love to hear from you.

Help build the intelligence behind the next generation of AI.