Senior LLM Training & Post-Training Engineer (Hybrid/Remote)

Lightning AI; San Francisco, Seattle, New York (CA, WA, NY); Hybrid; USD 165,000 - 310,000; Full time

Benefits offered by this job

Comprehensive Health Coverage

Meaningful Equity

401(k) matching & pension

Unlimited PTO

Hybrid work model

Job summary

Lightning AI is seeking a Senior Research Engineer focused on training and post-training of transformer-based language models. You will advance how LLMs are trained, fine-tuned, evaluated, and deployed across the platform, collaborating with researchers, infrastructure engineers, and customers.

You should excel at turning cutting-edge research into production systems, with deep experience in model training and AI systems engineering.

Qualifications

• Significant experience training and optimizing transformer-based models with PyTorch.
• Experience with continued pretraining, SFT, RLHF, and reward modeling techniques.
• Strong distributed training knowledge for multi-GPU systems.
• Production-quality Python software and research tooling experience.
• Experience designing experiments and debugging complex training issues.

Responsibilities

• Design, build, and optimize LLM training and post-training pipelines.
• Improve model quality via supervised fine-tuning and evaluation experiments.
• Develop PyTorch-based training infrastructure and developer workflows.
• Optimize distributed training for throughput and GPU utilization.
• Collaborate with researchers, infra, and customers to translate learnings into platform improvements.
• Contribute to open-source projects and documentation.

Skills

LLM training

Post-training techniques

Distributed training

Python software

Experiment design

Collaboration

Fast-moving environments

Advanced degree

Education

Master’s degree or PhD in ML/CS

Tools

DeepSpeed

Megatron-LM

Hugging Face Transformers

TRL/PEFT

Job description

Lightning AI is seeking a Senior Research Engineer focused on training and post-training of transformer-based language models. You will advance how LLMs are trained, fine-tuned, evaluated, and deployed across the platform, collaborating with researchers, infrastructure engineers, and customers.

You should excel at turning cutting-edge research into production systems, with deep experience in model training and AI systems engineering.