Pretraining Engineer — Scalable Language Models on GPUs

Zyphra Technologies Inc.

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Comprehensive medical, dental, vision
Relocation support
In-office snacks and meals
Unlimited PTO
Team in San Francisco

Job summary

Zyphra Technologies Inc. in San Francisco seeks a Research Engineer - Language Model Pre-Training to shape our model roadmap through end-to-end pretraining development. You will work closely with the pretraining team to translate insights into next-gen models.

You will lead large-scale training runs, optimize performance of the pretraining stack, collect and evaluate datasets, and research architectures and methodologies, including optimizer ablations.

Qualifications

  • Deep expertise in solving machine learning problems and training models.
  • Experience with training on large-scale (multi-node) GPU clusters.
  • Deep understanding of model training pipelines – including model/data parallelism, distributed optimizers, etc.
  • Strong grasp of experimental methodology for running rigorous ablations and hypothesis testing.
  • Understanding of large-scale, highly parallel data processing pipelines.
  • High proficiency with PyTorch and Python.
  • Strong ability to dive into large pre-existing codebases and rapidly get up to speed.
  • Published machine learning research in well-respected venues is a plus.

Responsibilities

  • Large-scale training runs and model parallelization
  • Performance optimization of our pretraining stack
  • Dataset collection, processing, and evaluation
  • Architecture and methodology research, including optimizer ablations

Skills

Deep ML problem solving
Large-scale multi-node GPU training
Model training pipelines (data/model,
Ablations and experimental methodology
Large-scale parallel data processing
PyTorch
Python
Codebase navigation
Published ML research (nice to have)

Education

Postgraduate degree in Scientific field (CS/EE/Math/Physics)

Tools

PyTorch
Python

Job description

Zyphra Technologies Inc. in San Francisco seeks a Research Engineer - Language Model Pre-Training to shape our model roadmap through end-to-end pretraining development. You will work closely with the pretraining team to translate insights into next-gen models.

You will lead large-scale training runs, optimize performance of the pretraining stack, collect and evaluate datasets, and research architectures and methodologies, including optimizer ablations.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Architect, Language Model Pretraining
Architect, Language Model Pretraining

Bison • San Francisco (CA)

On-site
USD 190,000 - 240,000
Comprehensive medical, dental, vision,
FSA plans
Competitive compensation and 401(k)
+4
Research Engineer - Language Model Pre-Training
Research Engineer - Language Model Pre-Training

Bison • San Francisco (CA)

On-site
USD 190,000 - 240,000
Comprehensive medical, dental, vision,
FSA plans
Competitive compensation and 401(k)
+4
Research Engineer – Language Model Pre-Training
Research Engineer – Language Model Pre-Training

Zyphra Technologies Inc. • San Francisco (CA)

On-site
USD 180,000 - 240,000
Comprehensive medical, dental, vision
Relocation support
In-office snacks and meals
+2
Vision & Vision-Language ML Research Engineer
Vision & Vision-Language ML Research Engineer

Bison • San Francisco (CA)

On-site
USD 140,000 - 230,000
Medical, dental, vision
401(k) plan
Relocation support
+3
Innovative Model Architectures Research Engineer
Innovative Model Architectures Research Engineer

Bison • San Francisco (CA)

On-site
USD 150,000 - 230,000
Comprehensive medical, dental, vision,
FSA plans
Competitive compensation and 401(k)
+4
AI Performance & Kernel Optimization Engineer
AI Performance & Kernel Optimization Engineer

Bison • San Francisco (CA)

On-site
USD 180,000 - 260,000
Medical benefits
Dental & Vision
FSA plans
+5
Vision & Vision-Language ML Research Engineer
Vision & Vision-Language ML Research Engineer

Zyphra • San Francisco (CA)

On-site
USD 150,000 - 210,000
Medical benefits
Dental plan
Vision plan
+4
Model Architecture Research Engineer - Innovate & Prototype
Model Architecture Research Engineer - Innovate & Prototype

Zyphra Technologies Inc. • San Francisco (CA)

On-site
USD 150,000 - 230,000
Relocation support
Immigration support
Unlimited PTO
+1
Frontier AI Kernel & Performance Engineer
Frontier AI Kernel & Performance Engineer

Zyphra Technologies Inc. • San Francisco (CA)

On-site
USD 180,000 - 260,000
Medical, dental, vision, and FSA plans
401(k) plan
Relocation and immigration support
+3
Research Engineer - Model Architectures
Research Engineer - Model Architectures

Bison • San Francisco (CA)

On-site
USD 150,000 - 230,000
Comprehensive medical, dental, vision,
FSA plans
Competitive compensation and 401(k)
+4