On-Device AI Inference Engineer — Low-Latency, Embedded

Hark; San Jose (CA); On-site; USD 200,000 - 450,000; Full time

Job summary

Hark is seeking an experienced systems engineer to optimize transformer workloads on dedicated hardware, writing kernels and runtime paths for latency and power targets. You will push models from full precision toward INT8/INT4 while coordinating with hardware teams to maximize efficiency.

The role requires 4–8+ years in performance-critical software, strong C/C++, SIMD, and deep knowledge of attention/KV-cache dynamics within deployment constraints.

Qualifications

• 4-8+ years writing performance-critical software and hands-on optimization.
• Strong C/C++ and familiarity with SIMD, custom kernels, memory layout.
• Understanding of attention, KV-cache behavior, and transformer inference timing.
• Experience optimizing software for constrained hardware (GPUs/NPUs/DSPs).

Responsibilities

• Write and optimize low-level kernels and runtime paths for transformer workloads on target silicon.
• Manage memory residency, scheduling, and swap behavior across concurrent models.
• Profile models on real hardware and eliminate bottlenecks to close theory-to-performance gaps.
• Convert full-precision models to INT8/INT4 within size, latency, and power budgets.
• Collaborate with hardware teams to enable new accelerators and feed deployment constraints back to model teams.
• Ensure transformer workloads run efficiently on hardware accelerators as they come online.

Skills

C/C++

SIMD

Performance optimization

Profiling

Transformer inference

Tools

ONNX Runtime

TVM

MLIR

TensorRT

Job description

Hark is seeking an experienced systems engineer to optimize transformer workloads on dedicated hardware, writing kernels and runtime paths for latency and power targets. You will push models from full precision toward INT8/INT4 while coordinating with hardware teams to maximize efficiency.

The role requires 4–8+ years in performance-critical software, strong C/C++, SIMD, and deep knowledge of attention/KV-cache dynamics within deployment constraints.