Senior Runtime Architect for AI Inference Hardware

Normal Computing Corporation; Palo Alto (CA); On-site; USD 180,000 - 280,000; Full time

Job summary

Normal Computing is seeking a senior runtime engineer to own the design and implementation of its AI runtime. You will work with simulators and FPGA prototypes, guiding how software interfaces with future silicon while remaining stable after tape-out.

You will lead cross-team work with the compiler and firmware teams, optimize memory management, and push for low-latency LLM inference on memory-bound hardware.

Qualifications

• Experience building runtimes, drivers, or hardware abstraction layers for accelerators.
• Strong systems programming in C++ or Rust: memory management, concurrency, and low-level performance.
• Track record of owning a complex systems component end to end, from design through production, and driving technical decisions across teams.
• Understanding of LLM inference workloads and what they demand of a runtime.
• Comfort building software for hardware that is still evolving, from simulation through bring-up.

Responsibilities

• Own runtime design including device management, memory allocation, executable loading, and command submission.
• Define the boundary with the compiler above the runtime and with the driver and firmware below it.
• Manage the state that lives on hardware between requests, such as resident model weights.
• Keep runtime overhead off the critical path for LLM inference and optimize latency/throughput.
• Provide technical leadership and review designs that depend on the runtime across teams.

Skills

Runtimes/ Drivers

C++/Rust

End-to-end ownership

LLM inference

Hardware bring-up

CUDA/ROCm

Tools

CUDA/ROCm

Job description

Normal Computing is seeking a senior runtime engineer to own the design and implementation of its AI runtime. You will work with simulators and FPGA prototypes, guiding how software interfaces with future silicon while remaining stable after tape-out.

You will lead cross-team work with the compiler and firmware teams, optimize memory management, and push for low-latency LLM inference on memory-bound hardware.