ML Training Performance Engineer - On-Site in Salt Lake City

Blackrock-Neurotech; Salt Lake City (UT); On-site; USD 140,000 - 210,000; Full time

Job summary

Blackrock Neurotech is seeking an ML Training Performance Engineer to own the efficiency and scalability of neural network training on GPU and cloud infrastructure. You will work as an individual contributor on a small research team across the training stack, partnering with model researchers, data engineers, and IT to improve performance while preserving scientific integrity.

The role is on-site at our Salt Lake City headquarters.

Qualifications

• Bachelor’s degree or equivalent practical experience with 3+ years in industry
• Demonstrated performance improvements in large DL training workloads
• Measurement-driven approach with profiling/benchmarking
• Strong Python and C++ programming and debugging
• Deep understanding of GPU execution and memory hierarchies
• Experience with CUDA or HIP/ROCm kernels
• Proficiency in PyTorch and mixed-precision training
• Strong understanding of DL architectures and training computations
• Experience with distributed training and GPU interconnects
• Experience configuring Linux-based GPU environments and containers

Responsibilities

• Own training performance across single-GPU, multi-GPU, and multi-node workloads with reproducible baselines
• Profile full training path to identify bottlenecks and prioritize improvements
• Optimize tensor layouts, memory, and execution graphs for larger models
• Develop and validate CUDA/HIP kernels when needed for performance
• Improve Python training scripts, framework settings, batching, and optimizers
• Design and tune distributed training strategies based on model structure and memory
• Collaborate with researchers on hardware-aware architecture and hyperparameters
• Coordinate with infrastructure engineers on prefetching and data I/O overlap
• Work with IT on GPUs, cloud instances, networking, and containers
• Build robust checkpoint/restart/recovery workflows and regression checks
• Communicate benchmark results and resource recommendations clearly

Skills

Python

C++

GPU perf optimization

PyTorch

Distributed training

Profiling

Linux

Collaboration

CUDA

Education

Bachelor’s degree in CS/CE or related field

Master’s degree in related field

PhD in related field

Tools

CUDA

HIP/ROCm

Triton

Gloo/NCCL

Docker

Job description

Blackrock Neurotech is seeking an ML Training Performance Engineer to own the efficiency and scalability of neural network training on GPU and cloud infrastructure. You will work as an individual contributor on a small research team across the training stack, partnering with model researchers, data engineers, and IT to improve performance while preserving scientific integrity.

The role is on-site at our Salt Lake City headquarters.