Software Engineer, Inference Runtime

EngRadar; New York (NY); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Equity grants

Medical plan

Vision plan

Dental plan

Catered team lunch

Expensed dinners in the office

Flexible PTO

Flexible WFH

Job summary

EngRadar seeks a senior ML infrastructure engineer to develop and optimize the inference stack for both on-device and cloud AI, integrating new engines and multimodal models. You will work on latency and throughput improvements across CPU/GPU runtimes and contribute to open-source projects.

Required is extensive experience building production ML systems with strong Python and C++ skills, plus deep transformer knowledge and CPU/GPU profiling expertise.

Qualifications

• Proven experience building production ML systems or performance-sensitive infrastructure.
• Strong proficiency in Python and C++.
• Deep knowledge of transformer architectures and profiling CPU/GPU workloads.

Responsibilities

• Develop and optimize the inference stack for on-device and cloud AI.
• Integrate new engines and multimodal models into the stack.
• Improve latency and throughput across hardware runtimes and contribute to open-source projects.

Skills

Python

C++

Transformer Architectures

Model Inference

CPU Profiling

GPU Profiling

PyTorch

Llama.cpp

MLX

ExecuTorch

vLLM

SGLang

TensorRT-LLM

CUDA

Metal

Vulkan

Tools

PyTorch

Llama.cpp

CUDA

Vulkan

Job description

The role involves developing and optimizing the inference stack for on-device and cloud AI, integrating new engines and multimodal models. Responsibilities include improving latency and throughput across various hardware runtimes and contributing to open-source projects.

Requirements: Candidates need significant experience building production ML systems or performance-sensitive infrastructure with strong proficiency in Python and C++. Deep knowledge of transformer architectures and experience profiling CPU/GPU workloads are essential.

Key Skills: Python, C++, Transformer Architectures, Model Inference, CPU Profiling, GPU Profiling, PyTorch, Llama.cpp, MLX, ExecuTorch, vLLM, SGLang, TensorRT-LLM, CUDA, Metal, Vulkan

Benefits: Competitive salary, Equity grants, Medical healthcare plan, Vision healthcare plan, Dental healthcare plan, Catered team lunch, Expensed dinners in the office, Flexible PTO, Flexible WFH