Research Engineer, Large Language Models

Immiland Canada

California

Hybrid

USD 170,000 - 210,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Immiland Canada is hiring a Research Engineer to design, run, and scale training experiments on thousands of GPUs, turning research into production-grade models. You will join a small team owning data, pre-training, post-training, and evaluation throughout the model lifecycle.

You will work with alignment, infrastructure, and product teams, implementing ideas from papers, building robust data pipelines, and ensuring rigorous, reproducible results at scale.

Qualifications

  • 3+ years of hands-on DL training or a related PhD.
  • Strong Python skills and PyTorch or JAX expertise.
  • Experience with Transformer architectures and scalable training.
  • Ability to read research papers and implement ideas from scratch.
  • Rigorous experiments, baselines, and reproducible results.

Responsibilities

  • Design and run large-scale pre-training and fine-tuning experiments, from small ablations to full training runs.
  • Study scaling laws to choose model size, data mix, and hyperparameters under a fixed compute budget.
  • Improve training efficiency through parallelism strategies (data, tensor, pipeline, FSDP), mixed precision, and custom CUDA or Triton kernels.
  • Diagnose and fix training instabilities such as loss spikes, divergence, and hardware failures.
  • Build and improve data pipelines, including filtering, deduplication, quality scoring, and dataset mixing.
  • Implement ideas from recent research papers and test whether they hold at scale.
  • Build evaluations that measure real model capabilities and track results rigorously.
  • Work closely with the alignment, infrastructure, and product teams, and share findings through clear write-ups.

Skills

Python
PyTorch/JAX
Transformer models
Distributed training
Experiment tracking
Communication skills

Education

PhD in CS/Math/Physics

Tools

DeepSpeed
Megatron-LM
CUDA
Triton
Slurm
Kubernetes

Job description

About the Role

We are hiring a Research Engineer to help train our next generation of large language models. You will design, run, and scale training experiments on thousands of GPUs, and turn research ideas into production-grade models.

You will join a small team of researchers and engineers who own the full model lifecycle: data, pre-training, post-training, and evaluation. Research and engineering are not separate here: you will write the code, run the experiments, and shape the conclusions.


Key Responsibilities – What You'll Do
  • Design and run large-scale pre-training and fine-tuning experiments, from small ablations to full training runs.

  • Study scaling laws to choose model size, data mix, and hyperparameters under a fixed compute budget.

  • Improve training efficiency through parallelism strategies (data, tensor, pipeline, FSDP), mixed precision, and custom CUDA or Triton kernels.

  • Diagnose and fix training instabilities such as loss spikes, divergence, and hardware failures.

  • Build and improve data pipelines, including filtering, deduplication, quality scoring, and dataset mixing.

  • Implement ideas from recent research papers and test whether they hold at scale.

  • Build evaluations that measure real model capabilities and track results rigorously.

  • Work closely with the alignment, infrastructure, and product teams, and share findings through clear write-ups.

Required Qualifications – What We're Looking For
  • 3+ years of hands-on experience training deep learning models, or a PhD in a related field (computer science, mathematics, or physics).

  • Strong Python skills and deep expertise in PyTorch or JAX.

  • Practical experience with Transformer architectures, including attention, tokenization, embeddings, and positional encodings.

  • Experience with distributed training on multi-node GPU clusters (FSDP, DeepSpeed, Megatron-LM, or similar).

  • Solid foundations in linear algebra, probability, statistics, and optimization.

  • Ability to read a research paper, implement it from scratch, and judge whether it works.

  • Rigorous experimental habits, including clean baselines, ablations, reproducible results, and experiment tracking.

  • Clear written and verbal communication skills, and comfort working in a fast-moving, ambiguous environment.

Preferred Qualifications – Nice to Have
  • Publications at top venues (NeurIPS, ICML, ICLR, ACL) or well-known open-source contributions.

  • Experience pre-training an LLM from scratch, at any scale.

  • Post-training experience, including supervised fine-tuning, RLHF, DPO, or other preference-optimization methods.

  • GPU performance work involving CUDA, Triton, FlashAttention-style kernels, and profiling.

  • Experience with efficient inference, including quantization, distillation, and optimized model serving.

  • Cluster and infrastructure experience with Slurm, Kubernetes, or cloud GPU platforms.

  • Large-scale data engineering experience (Spark, Ray, dataset curation for multilingual or domain-specific corpora).


Thank you for your interest!

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Research Engineer
Research Engineer

Vidman AI • United States

Remote
USD 140,000 - 230,000
Research Staff, LLMs [EMEA - Remote]
Research Staff, LLMs [EMEA - Remote]

Deepgram • United States

Remote
USD 170,000 - 210,000
Member of Technical Staff - ML Systems
Member of Technical Staff - ML Systems

NextToken AI Inc. • Mountain View (CA)

On-site
USD 180,000 - 250,000
Research Engineer, ML Infrastructure
Research Engineer, ML Infrastructure

cognition • San Francisco (CA)

On-site
USD 180,000 - 250,000
ML Research Engineer, Training
ML Research Engineer, Training

Weave Robotics • San Francisco (CA)

On-site
USD 180,000 - 250,000
Member of Technical Staff — Training Infrastructure
Member of Technical Staff — Training Infrastructure

Kindredventures • San Francisco (CA)

On-site
USD 180,000 - 240,000
Research Scientist/Research Engineer, Midtraining
Research Scientist/Research Engineer, Midtraining

Periodic Labs • Menlo Park (CA)

On-site
USD 250,000 - 350,000
Research Engineer, Post-Training
Research Engineer, Post-Training

Collision • San Francisco (CA)

On-site
USD 120,000 - 180,000
Research Scientist / Engineer – Reinforcement Learning Infrastructure
Research Scientist / Engineer – Reinforcement Learning Infrastructure

lumaai • United States

Remote
USD 180,000 - 260,000
Member of Technical Staff, Model Training & Post-Training Model Training
Member of Technical Staff, Model Training & Post-Training Model Training

TensorScale AI, Inc. • San Francisco (CA)

On-site
USD 180,000 - 240,000