Machine Learning Performance Engineer (Inference)

Career Techniques; New York (NY); On-site; USD 200,000 - 300,000; Full time

Job summary

Career Techniques in New York, NY seeks a hands-on ML inference engineer to optimize production-grade models across CPUs, GPUs, and FPGAs. You will lead platform evaluations, tune kernels, and push memory and interconnect performance to meet latency requirements.

Collaborate with ML researchers, HPC and datacenter teams to deploy compact, low-latency inference workloads, using tools like Triton, TensorRT and Nsight, with a focus on reliability and scalable throughput.

Qualifications

• 2+ years optimizing DL inference in latency-sensitive environments.
• Deep expertise in PyTorch/JAX with Python/C++.
• Experience in custom GPU kernel development and libraries for performance.
• Strong knowledge of GPU architecture and memory hierarchies.
• Experience benchmarking across heterogeneous compute architectures.

Responsibilities

• Lead evaluation of inference platforms across CPUs, GPUs, and FPGAs.
• Analyze and optimize execution across deep memory hierarchies.
• Collaborate with infrastructure teams on latency-critical strategy.
• Develop highly optimized GPU kernels and integrate libraries.
• Implement model compression for low-latency inference.
• Work with ML researchers, HPC/FPGA/datacenter teams on deployments.

Skills

PyTorch

JAX

Python

C++

Mixed-precision

GPU kernel development

TensorRT

Profiling tools

CuBLAS/CUTLASS

FPGAs/ASICs

Tools

Triton

TensorRT

ONNX

IREE

HLS4ML

cuBLAS

CUTLASS

Nsight Systems

Nsight Compute

Job description

Responsibilities

• Benchmarking & Strategy:
  • Lead the technical evaluation of diverse inference platforms - ranging across CPUs, GPUs, and FPGAs - to guide infrastructure deployment decisions.
• System Architecture Optimization:
  • Analyze and enhance execution across deep memory hierarchies to maximize resource utilization and parallel processing. You will assess and resolve memory subsystem and interconnect bottlenecks across the end-to-end inference lifecycle.
• Infrastructure & Deployment Feasibility:
  • Collaborate with Infrastructure teams to understand thermal, power, and operational constraints of hardware platforms to design inference strategies for our latency-critical trading strategies that fit within those envelopes.
• GPU Kernel Development:
  • Develop highly optimized kernels and integrate specialized performance libraries to extract maximum computational throughput from the underlying silicon.
• Model Optimization & Deployment:
  • Implement advanced model reduction techniques (quantization, pruning, distillation) to ensure compact memory footprints and numerical stability. Prioritize optimization for low-latency, event-level inference workloads to meet real-time trading requirements.
• Cross-Functional Collaboration:
  • Collaborate closely with ML Researchers, HPC Engineers, FPGA Engineers, and Datacenter Engineers to bring to fruition target deployments.

Qualifications

• 2+ years of experience optimizing deep learning inference in latency-sensitive or high-throughput production environments, in any domain.
• ML Frameworks: Deep expertise in lower-level ML framework development (PyTorch/JAX), paired with strong Python/C++ skills and a thorough understanding of mixed-precision computation.
• Kernel Development & Optimization Tooling: Proven experience in custom GPU kernel development. Deep familiarity with advanced optimization libraries and compilers (e.g., Triton, TensorRT, ONNX, IREE, HLS4ML, cuBLAS, CUTLASS) as well as profiling tools (e.g., Nsight Systems, Nsight Compute).
• GPU Architecture Mastery: Deep expertise in GPU microarchitecture, encompassing SM execution, warp scheduling, and full memory hierarchy optimization (registers to HBM).
• Cross-Architecture Benchmarking: Proven record of rigorous, data-driven approach to evaluating inference performance across heterogeneous compute architectures.
• Bonus: Practical experience targeting and optimizing inference workloads on specialized hardware ecosystems, including FPGAs and ASICs.
• Prior experience in financial trading is not required.

Comp: $200-300K + Bonus