Technical Lead, On-Device AI Inference San Jose

Hark; San Jose, Northern (CA, KY); Hybrid; USD 300,000 - 500,000; Full time

Job summary

Hark is seeking a senior hardware/software leader to own how our models run on silicon. You will select accelerators, shape architectures within latency, memory, and power budgets, and build the low-level inference stack that enables millisecond responses on battery-powered devices.

You will lead a team of engineers, partner with silicon vendors, and drive efficient transformer execution on new accelerators from research to product.

Qualifications

• 8–12+ years in high-performance computing with production workloads on GPUs or accelerators.
• Deep understanding of attention, KV-cache, quantization, and memory bandwidth limits.
• You've designed or optimized inference engines, runtimes, or ML compilers and write kernels when needed.
• You've led teams on performance-critical software and set architectural direction.

Responsibilities

• Evaluate GPUs, NPUs, DSPs, and accelerators for on-device deployment and advocate for hardware decisions.
• Collaborate with foundation model and audio ML teams to meet deployment constraints before training.
• Build low-level execution layer, kernels, runtimes, and compiler paths for transformer workloads on target hardware.
• Partner with silicon vendors and internal teams to bring up accelerators and optimize transformer execution.
• Hire and lead engineers, setting the technical bar for the inference stack.

Skills

High-performance computing

Team leadership

ML inference

GPU/accelerator knowledge

Performance optimization

Tools

TensorRT

ONNX Runtime

TVM

MLIR

Job description

Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persistent memory.

We're pairing that intelligence with next-generation hardware to create a universal interface between humans and machines. While today's AI largely operates through chat boxes and decade-old devices, Hark is focused on what comes next: agentic systems that interact naturally with people and the real world.

To get there, we're developing multimodal models and next-generation AI hardware together - designed from the ground up as a single, unified interface for a new era of intelligent systems.

About the Role

You'll own how Hark's models run on the silicon we ship: selecting the accelerators our devices are built around, co-designing architectures against real latency, memory, and power budgets, and building the low-level inference stack that turns a trained model into something that responds in milliseconds on a battery. You'll build and lead the team that does it. The ceiling on what our hardware can do is set here.

Responsibilities

• Evaluate GPUs, NPUs, DSPs, and specialized accelerators for on-device deployment, and own the recommendation hardware decisions are made against.
• Work with the foundation model and audio ML teams to shape architectures that meet deployment constraints before training locks them in.
• Build the low-level execution layer, custom kernels, runtime systems, and compiler paths that transformer workloads run through on target hardware.
• Partner with silicon vendors and internal hardware teams to bring up new accelerators and get efficient transformer execution on them early.
• Hire and lead a team of engineers on performance-critical software, and set the technical bar for the inference stack.

Requirements

• 8–12+ years in high-performance computing, including production workloads deployed on GPUs, NPUs, or specialized accelerators.
• Deep understanding of attention, KV-cache behavior, quantization effects, and memory bandwidth limits.
• You've designed or optimized inference engines, distributed runtimes, or ML compilers, and you write the kernels yourself when it matters.
• Experience leading teams on performance-critical software. You've set direction on a stack, not just contributed to one.
• You've taken a model from a research checkpoint to running on constrained hardware in a product people use.

Bonus Qualifications

• Hands-on experience with Hexagon DSP, Ambiq-class MCUs, or comparable embedded AI silicon.
• Experience with speech, audio, or streaming multimodal inference where latency is perceptible to the user.
• Contributions to open-source inference or compiler toolchains (TensorRT, ONNX Runtime, TVM, MLIR, and similar).

Compensation

The US base salary range for this full-time position is between $300,000 - $500,000 annually.

The pay offered for this position may vary based on several individual factors, including job-related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.