On-Device AI Inference Engineer — Ultra-Low Latency

Hark; San Jose, Northern (CA, KY); Hybrid; USD 200,000 - 450,000; Full time

Job summary

Hark is building multimodal AI hardware and software designed to run transformer workloads with low latency and power constraints. You will write and optimize kernels, manage memory and scheduling across models, and profile real hardware to close the gap between theory and practice.

Ideal candidates have 4-8+ years in performance-critical software for accelerators, strong C/C++, and experience with ONNX Runtime, TVM, MLIR or TensorRT to push inference efficiency on constrained devices.

Qualifications

• 4-8+ years writing performance-critical software with hands-on optimization on GPUs, NPUs, DSPs, or similar accelerators.
• Strong C/C++ and comfort with SIMD, custom kernels, memory layout, and the profiling tools that go with them.
• You understand attention, KV-cache behavior, and where transformer inference actually spends its time and memory bandwidth.
• You reason in compute, memory, and power budgets, and you've optimized against them rather than around them.
• You've had a model you optimized run in a product on constrained hardware.

Responsibilities

• Write and optimize the low-level kernels and runtime paths that transformer workloads execute through on target silicon.
• Decide how multiple models share limited memory and power i.e. residency, scheduling, and swap behavior across concurrent workloads.
• Profile models on real hardware, find the bottlenecks, and close the gap between theoretical and delivered performance.
• Take models from full precision to INT8/INT4 and get them running within per-product size, latency, and power budgets.
• Get transformer workloads executing efficiently on new accelerators as they come online, working alongside the hardware team.
• Feed real deployment constraints back to the model teams so architecture decisions account for what the hardware can actually do.

Skills

C/C++

Performance optimization

Profiling tools

Memory layout

Compiler/MLIR

Tools

ONNX Runtime

TVM

MLIR

TensorRT

Job description

Hark is building multimodal AI hardware and software designed to run transformer workloads with low latency and power constraints. You will write and optimize kernels, manage memory and scheduling across models, and profile real hardware to close the gap between theory and practice.

Ideal candidates have 4-8+ years in performance-critical software for accelerators, strong C/C++, and experience with ONNX Runtime, TVM, MLIR or TensorRT to push inference efficiency on constrained devices.