Runtime Engineer

Normal Computing; New York (NY); On-site; USD 180,000 - 260,000; Full time

Job summary

Normal Computing is building a memory-centric runtime for large-model inference. You will own its design and set its technical direction, working with simulators and FPGA prototypes before silicon arrives.

This senior, hands-on role focuses on device management, memory allocation, executable loading, and cross-team interfaces, with emphasis on low-latency, high-throughput LLM workloads.

Qualifications

• Experience building runtimes, drivers, or hardware abstraction layers for accelerators.
• Strong systems programming in C++ or Rust with memory management and concurrency.
• Track record owning a complex systems component end to end.
• Understanding of LLM inference workloads and runtime demands.
• Experience with simulation and bring-up for evolving hardware.

Responsibilities

• Design the runtime: device management, memory allocation, and transfers.
• Hide hardware differences behind a stable interface for software.
• Manage resident model weights across requests and ongoing state.
• Minimize runtime overhead on latency and throughput for LLMs.
• Set runtime direction and collaborate with compiler, firmware, and driver teams.

Skills

Systems programming

C++

Rust

Concurrency

Hardware abstraction

Tools

Simulators

FPGA

CUDA

ROCm

Job description

Normal's hardware runs the most memory-bound parts of large-model inference inside memory itself, rather than shuttling weights to a processor. The software that makes this programmable doesn't exist yet in any standard form. The runtime is the layer the rest of that software runs through: it is how compiled programs get loaded, scheduled, and executed on our hardware. You will own its design and much of its implementation, and set its technical direction.

This is a senior, hands-on role. Our chip is pre-silicon, so you will build the runtime against our simulators and FPGA prototypes first, behind one interface, so that software above the runtime doesn't change when silicon arrives.

What You Will Own

• Runtime Architecture: Own the design of the runtime: device management, memory allocation and host-device transfer, executable loading, and command submission. Success means the compiler and serving layers program our hardware through one stable interface.
• Hardware Abstraction: Hide the differences between simulators, FPGA prototypes, silicon, and later hardware generations behind that interface.
• Device State: Manage the state that lives on our hardware between requests, such as resident model weights, so the serving layer can rely on it.
• Performance: Keep runtime overhead off the critical path for LLM inference, measured by end-to-end latency and throughput.
• Cross-Team Interfaces: Define the boundary with the compiler above the runtime, and with the driver and firmware below it, which our hardware team owns.
• Technical Leadership: Set the runtime's technical direction, review designs that depend on it, and work with compiler and serving engineers so the interface serves them.

What Makes You a Great Fit

• Experience building runtimes, drivers, or hardware abstraction layers for accelerators or other non-standard hardware
• Strong systems programming in C++ or Rust: memory management, concurrency, and low-level performance
• Track record of owning a complex systems component end to end, from design through production, and driving technical decisions across teams
• Understanding of LLM inference workloads and what they demand of a runtime
• Comfort building software for hardware that is still evolving, from simulation through bring-up

Bonus Points

• Experience with open runtime designs such as IREE's HAL, or with CUDA or ROCm runtime internals
• Experience with in-memory compute or other non-von-Neumann hardware
• Experience integrating a runtime with an inference server such as vLLM
• Contributions to open-source runtime or systems infrastructure