GPU-Driven ML Training Performance Engineer

Socket.dev; Salt Lake City (UT); On-site; USD 150,000 - 200,000; Full time

Job summary

Blackrock Neurotech is seeking an ML Training Performance Engineer to own and optimize the efficiency and scalability of model training on GPU and cloud infra in Salt Lake City. You will work across the training stack with researchers and data/infra teams to drive end-to-end performance gains, maintaining numerical correctness.

You will profile, optimize, and validate training workloads, develop kernels, and tune distributed strategies to fit larger models within available resources.

Qualifications

• Proven performance improvements for large DL training workloads.
• Experience with profiling and benchmarking to validate end-to-end gains.
• Strong Python and C++ programming and debugging skills.
• Bachelor-level degree or equivalent practical experience.

Responsibilities

• Own training performance across single/multi-GPU and multi-node workloads.
• Profile full training paths to identify bottlenecks and quantify impact.
• Tune tensor layouts, memory, precision, and activation checkpointing.
• Develop and validate custom GPU kernels with CUDA, Triton, or HIP/ROCm.
• Improve training scripts, batching, and optimizer execution without changing behavior.
• Design distributed training strategies based on model structure and interconnects.
• Collaborate with researchers on hardware-aware changes and convergence.
• Coordinate data delivery with prefetching and I/O overlap.
• Work with IT on GPU selection, cloud configs, and capacity planning.

Skills

Python

C++

GPU understanding

CUDA

PyTorch

Distributed training

Profiling & benchmarking

Performance optimization

System debugging

Tensor concepts

Education

Bachelor's degree in CS/CE/EE or related field

Tools

CUDA

HIP/ROCm

Triton

Job description

Blackrock Neurotech is seeking an ML Training Performance Engineer to own and optimize the efficiency and scalability of model training on GPU and cloud infra in Salt Lake City. You will work across the training stack with researchers and data/infra teams to drive end-to-end performance gains, maintaining numerical correctness.

You will profile, optimize, and validate training workloads, develop kernels, and tune distributed strategies to fit larger models within available resources.