Head of On-Device AI Inference & Performance

Hark; San Jose (CA); On-site; USD 300,000 - 500,000; Full time

Job summary

Hark is an AI company building proactive, multimodal intelligence with advanced hardware and software integration. You will own how our models run on silicon, selecting accelerators, co-designing architectures against latency, memory, and power budgets, and building the low-level inference stack that enables millisecond responsiveness on battery-powered devices.

You will lead the team responsible for implementing efficient transformer execution on new accelerators, collaborating with silicon

Qualifications

• 8–12+ years in high-performance computing and production workloads on GPUs/NPUs/specialized accelerators.
• Deep understanding of attention, KV-cache, quantization, and memory bandwidth limits.
• Designed or optimized inference engines, distributed runtimes, or ML compilers, and wrote kernels when necessary.
• Experience leading teams on performance-critical software; set direction on a stack, not just contributed.
• Took a model from research checkpoint to running on constrained hardware in a product.

Responsibilities

• Evaluate GPUs, NPUs, DSPs, and accelerators for on-device deployment and advise hardware decisions.
• Collaborate with foundation model and audio ML teams to shape architectures meeting deployment constraints.
• Build the low-level execution layer, kernels, runtimes, and compiler paths for transformer workloads on target hardware.
• Partner with silicon vendors and internal hardware teams to bring up accelerators and optimize transformer execution.
• Hire and lead engineers on performance-critical software and set the bar for the inference stack.

Skills

High-performance computing

Attention & KV-cache

Inference engines / ML compilers

Leadership experience

Model deployment to hardware

Tools

TensorRT

ONNX Runtime

TVM

MLIR

Job description

Hark is an AI company building proactive, multimodal intelligence with advanced hardware and software integration. You will own how our models run on silicon, selecting accelerators, co-designing architectures against latency, memory, and power budgets, and building the low-level inference stack that enables millisecond responsiveness on battery-powered devices.

You will lead the team responsible for implementing efficient transformer execution on new accelerators, collaborating with silicon