On-Device AI Engineer: Edge Model Optimization

Deepgram; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

Deepgram’s embedded AI engineer role focuses on getting speech models running on on-device, low-power hardware across phones, wearables, and edge devices, with real-time, offline-capable capabilities.

You’ll optimize for latency, memory, power, and thermal budgets using quantization, pruning, distillation, and compilation, while collaborating with hardware and research teams to push edge-friendly architectures.

Qualifications

• Experience delivering production systems on resource-constrained hardware.
• Strong proficiency in C, C++, and/or Rust.
• Hands‑on experience with model optimization for on‑device deployment.
• Familiarity with edge inference runtimes (ONNX Runtime, TensorRT, TFLite, ExecuTorch) and/or vendor‑specific NPU/DSP toolchains.
• A strong understanding of hardware-software interaction — CPU/GPU/NPU/DSP architectures, memory hierarchies, fixed-point/integer arithmetic, and power management.
• Experience working close to the metal: bare‑metal or RTOS environments (e.g., FreeRTOS, Zephyr), embedded Linux, or microcontroller and edge SoC development.
• Strong communication skills and a builder mindset — you can scope an ambiguous optimization problem, drive it to a measurable result, and explain the tradeoffs clearly.

Responsibilities

• Take Deepgram’s Speech and Conversational models and get them running on embedded and low-power consumer hardware — defining the architecture for on-device, real-time inference across a diverse range of processors and accelerators.
• Optimize models for constrained targets through quantization, pruning, distillation, operator fusion, and architecture-specific compilation to meet strict latency, memory, power, and thermal budgets.
• Write and optimize performance-critical runtime code (C, C++, and/or Rust) for embedded environments, including bare-metal and real-time operating systems such as FreeRTOS and Zephyr.
• Integrate with industry-standard edge inference runtimes and vendor NPU/DSP toolchains, mapping model graphs efficiently onto on-device accelerators and CPU/GPU/NPU heterogeneity.
• Build the on-device runtime plumbing: model packaging, deployment pipelines, over-the-air update mechanisms, and lightweight telemetry for devices operating with limited or intermittent connectivity.
• Establish repeatable benchmarking and validation across target hardware — measuring latency, accuracy, power consumption, memory footprint, and resource utilization — and catch regressions before they ship.
• Partner with silicon and device vendors on SDK integration and performance tuning, getting our models to run efficiently on new chipsets and reference platforms.
• Collaborate with Research and Engine teams to influence model architectures toward edge-friendly designs from the start, reducing the optimization burden at deployment time.

Skills

C/C++/Rust

Communication skills

Builder mindset

Embedded systems

Tools

ONNX Runtime

TensorRT

TFLite

ExecuTorch

Job description

Deepgram’s embedded AI engineer role focuses on getting speech models running on on-device, low-power hardware across phones, wearables, and edge devices, with real-time, offline-capable capabilities.

You’ll optimize for latency, memory, power, and thermal budgets using quantization, pruning, distillation, and compilation, while collaborating with hardware and research teams to push edge-friendly architectures.