Inference Runtime Engineer - On-Device & Cloud AI, Flexible WFH

EngRadar; New York (NY); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Equity grants

Medical plan

Vision plan

Dental plan

Catered team lunch

Expensed dinners in the office

Flexible PTO

Flexible WFH

Job summary

EngRadar seeks a senior ML infrastructure engineer to develop and optimize the inference stack for both on-device and cloud AI, integrating new engines and multimodal models. You will work on latency and throughput improvements across CPU/GPU runtimes and contribute to open-source projects.

Required is extensive experience building production ML systems with strong Python and C++ skills, plus deep transformer knowledge and CPU/GPU profiling expertise.

Qualifications

• Proven experience building production ML systems or performance-sensitive infrastructure.
• Strong proficiency in Python and C++.
• Deep knowledge of transformer architectures and profiling CPU/GPU workloads.

Responsibilities

• Develop and optimize the inference stack for on-device and cloud AI.
• Integrate new engines and multimodal models into the stack.
• Improve latency and throughput across hardware runtimes and contribute to open-source projects.

Skills

Python

C++

Transformer Architectures

Model Inference

CPU Profiling

GPU Profiling

PyTorch

Llama.cpp

MLX

ExecuTorch

vLLM

SGLang

TensorRT-LLM

CUDA

Metal

Vulkan

Tools

PyTorch

Llama.cpp

CUDA

Vulkan

Job description

EngRadar seeks a senior ML infrastructure engineer to develop and optimize the inference stack for both on-device and cloud AI, integrating new engines and multimodal models. You will work on latency and throughput improvements across CPU/GPU runtimes and contribute to open-source projects.

Required is extensive experience building production ML systems with strong Python and C++ skills, plus deep transformer knowledge and CPU/GPU profiling expertise.