Low-Latency ML Inference Engineer

Career Techniques; New York (NY); Hybrid; USD 200,000 - 300,000; Full time

Job summary

Career Techniques in New York, NY seeks a hands-on ML inference engineer to optimize production-grade models across CPUs, GPUs, and FPGAs. You will lead platform evaluations, tune kernels, and push memory and interconnect performance to meet latency requirements.

Collaborate with ML researchers, HPC and datacenter teams to deploy compact, low-latency inference workloads, using tools like Triton, TensorRT and Nsight, with a focus on reliability and scalable throughput.

Qualifications

• 2+ years optimizing DL inference in latency-sensitive environments.
• Deep expertise in PyTorch/JAX with Python/C++.
• Experience in custom GPU kernel development and libraries for performance.
• Strong knowledge of GPU architecture and memory hierarchies.
• Experience benchmarking across heterogeneous compute architectures.

Responsibilities

• Lead evaluation of inference platforms across CPUs, GPUs, and FPGAs.
• Analyze and optimize execution across deep memory hierarchies.
• Collaborate with infrastructure teams on latency-critical strategy.
• Develop highly optimized GPU kernels and integrate libraries.
• Implement model compression for low-latency inference.
• Work with ML researchers, HPC/FPGA/datacenter teams on deployments.

Skills

PyTorch

JAX

Python

C++

Mixed-precision

GPU kernel development

TensorRT

Profiling tools

CuBLAS/CUTLASS

FPGAs/ASICs

Tools

Triton

TensorRT

ONNX

IREE

HLS4ML

cuBLAS

CUTLASS

Nsight Systems

Nsight Compute

Job description

Career Techniques in New York, NY seeks a hands-on ML inference engineer to optimize production-grade models across CPUs, GPUs, and FPGAs. You will lead platform evaluations, tune kernels, and push memory and interconnect performance to meet latency requirements.

Collaborate with ML researchers, HPC and datacenter teams to deploy compact, low-latency inference workloads, using tools like Triton, TensorRT and Nsight, with a focus on reliability and scalable throughput.