ML Training Performance Engineer - HPC & GPUs

towerresearchcapital

New York (NY)

On-site

USD 140,000 - 230,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Tower Research Capital in New York is seeking an experienced ML Training Engineer to bridge the gap between quantitative research and high-performance computing, building and optimizing the systems used to train machine learning models at scale.

You will accelerate the end-to-end training lifecycle—from data ingestion and distributed execution to kernel performance and hardware utilization—enabling researchers to iterate more quickly on increasingly complex models and datasets.

Qualifications

  • 3+ years of experience optimizing ML training workloads in high-performance, distributed, or large-scale computing environments.
  • Deep knowledge of ML frameworks such as PyTorch or JAX, including their execution models, compilation paths, autograd systems, and distributed-training capabilities.
  • Strong programming skills in Python and C++, with experience developing or optimizing performance-critical systems.
  • Proven experience with GPU kernel development and optimization using technologies such as CUDA, Triton, CUTLASS, cuBLAS, cuDNN, or related libraries.
  • Strong understanding of GPU architecture, including streaming multiprocessor execution, warp scheduling, tensor cores, and the memory hierarchy from registers through HBM.

Responsibilities

  • Benchmark model-training workloads across CPUs, GPUs, and other accelerator platforms to identify bottlenecks and guide Tower's compute infrastructure decisions.
  • Design and optimize distributed training strategies, including data, tensor, pipeline, and model parallelism.
  • Improve communication efficiency across multi-GPU and multi-node environments by optimizing collective operations, topology awareness, and computation-communication overlap.
  • Analyze and improve the full training pipeline, including data loading, preprocessing, memory management, forward and backward passes, optimizer execution, checkpointing, and experiment recovery.
  • Identify bottlenecks across compute, memory, storage, networking, and interconnects to increase accelerator utilization and researcher productivity.
  • Develop and optimize GPU kernels and performance-critical framework components for quantitative machine learning workloads.
  • Apply techniques such as mixed-precision training, gradient accumulation, activation checkpointing, operator fusion, and memory-efficient optimizers.
  • Partner with HPC and infrastructure teams to optimize workload scheduling, resource allocation, observability, fault tolerance, and reproducibility across shared compute environments.
  • Work closely with ML Researchers, Quantitative Researchers, HPC Engineers, Systems Engineers, and hardware specialists to translate research requirements into highly efficient training systems.

Job description

Tower Research Capital in New York is seeking an experienced ML Training Engineer to bridge the gap between quantitative research and high-performance computing, building and optimizing the systems used to train machine learning models at scale.

You will accelerate the end-to-end training lifecycle—from data ingestion and distributed execution to kernel performance and hardware utilization—enabling researchers to iterate more quickly on increasingly complex models and datasets.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Machine Learning Performance Engineer, Training
Machine Learning Performance Engineer, Training

towerresearchcapital • New York (NY)

On-site
USD 140,000 - 230,000
ML Inference Performance Engineer — Ultra-Low Latency
ML Inference Performance Engineer — Ultra-Low Latency

Priorslab • New York (NY)

Hybrid
USD 200,000 - 300,000
Hybrid working opportunities
Free breakfast, lunch, and snacks
Wellness reimbursement
+2
Senior Research Platform Engineer — HPC & ML
Senior Research Platform Engineer — HPC & ML

Tradermath • New York (NY)

Hybrid
USD 200,000 - 300,000
Generous paid time off
Hybrid working opportunities
Free breakfast, lunch, and snacks
+4
ML Engineer: Large-Scale Training & Real-Time Inference
ML Engineer: Large-Scale Training & Real-Time Inference

IMC Financial Markets • New York (NY)

On-site
USD 175,000 - 250,000
Discretionary bonus
Paid leave
Insurance
Applied ML Research Engineer - Benchmark & Scale ML Stack
Applied ML Research Engineer - Benchmark & Scale ML Stack

Tradermath • New York (NY)

Hybrid
USD 200,000 - 300,000
Generous paid time off
Hybrid working opportunities
Free breakfast, lunch, snacks
ML Research Platform Engineer
ML Research Platform Engineer

Tower Research Capital • New York (NY)

Hybrid
USD 200,000 - 300,000
Hybrid working opportunities
Generous PTO
Free meals in office
GPU Systems Engineer - Low-Latency HPC & AI Clusters
GPU Systems Engineer - Low-Latency HPC & AI Clusters

Tower Research Capital • New York (NY)

On-site
USD 200,000 - 300,000
Generous paid time off policies
Hybrid working opportunities
Free breakfast, lunch & snacks
+4
Research Platform Engineer — Hybrid ML/HPC & Multi-Tenant
Research Platform Engineer — Hybrid ML/HPC & Multi-Tenant

Tower Research Capital • New York (NY)

On-site
USD 200,000 - 300,000
Generous paid time off policies
Savings plans and other financial welI
Hybrid working opportunities
+6
ML Performance Engineer: Production GPU Optimizer
ML Performance Engineer: Production GPU Optimizer

Goliath Partners Inc. • New York (NY)

On-site
USD 200,000 - 300,000
ML Systems Engineer: Scalable Training & Inference
ML Systems Engineer: Scalable Training & Inference

IMC Trading • New York (NY)

On-site
USD 175,000 - 250,000