Inference Software Engineer

Etched; Cupertino (CA); On-site; Full time

Benefits offered by this job

Full medical, dental, and vision packages with premium covered

Housing subsidy of $2,000/month for nearby residents

Daily lunch and dinner in the office

Relocation support for those moving to Cupertino

Job summary

A pioneering AI hardware company in Cupertino is seeking a software engineer with experience in C++ and Python. You will design and implement high-performance software stacks that support advanced AI models. The ideal candidate has a strong background in distributed systems and a collaborative mindset. Join us to work on cutting-edge technology in a fully in-person team environment.

Qualifications

• Experience with C++ and Python.
• Familiarity with transformer model architectures and inference serving stacks.
• Experience working cross-functionally in large software and hardware organizations.

Responsibilities

• Contribute to the architecture and design of the Sohu host software stack.
• Implement high-performance, modular code across the software stack.
• Interface with firmware and drivers teams delivering HW/SW stack.

Skills

C++

Python

Familiarity with transformer model architectures

Experience working cross-functionally

Tools

Rust

CUDA compilation stack

Job description

Overview

Etched is building AI chips that are hard-coded for individual model architectures. Our first product (Sohu) supports transformers, delivering an order of magnitude more throughput and lower latency than GPUs. With Etched ASICs, you can build products that would be impossible with GPUs, like real-time video generation models and extremely deep and parallel chain-of-thought reasoning agents.

Key responsibilities

• Contribute to the architecture and design of the Sohu host software stack
• Implement high-performance, modular code across the complete Etched software stack, consisting of a mix of Rust, C++ and Python.
• Interface with firmware and drivers teams delivering highest-performance HW/SW stack.
• Work with AI model researchers and product-facing teams building out the Etched serving front-end.

Representative projects

• Build scheduling logic for handling continuous batching and real time inference
• Implement inference-time acceleration techniques such as speculative decoding, tree search, KV cache sharing, etc.
• Implement distributed networking primitives for efficient multi-server inference

You may be a good fit if you have

• Experience with C++ and Python
• Familiarity with transformer model architectures and inference serving stacks (vLLM, SGLang, etc.) or experience working in distributed inference/training environments
• Experience working cross-functionally in large software and hardware organizations

Strong candidates may also have

• Experience with Rust
• Familiarity with GPU kernels, the CUDA compilation stack and related tools, or other hardware accelerators
• Understanding of distributed systems, networking, and parallel programming
• Full medical, dental, and vision packages, with 100% of premium covered
• Housing subsidy of $2,000/month for those living within walking distance of the office
• Daily lunch and dinner in our office
• Relocation support for those moving to Cupertino

How we’re different

Etched believes in the Bitter Lesson. We think most of the progress in the AI field has come from using more FLOPs to train and run models, and the best way to get more FLOPs is to build model-specific hardware. Larger and larger training runs encourage companies to consolidate around fewer model architectures, which creates a market for single-model ASICs.

We are a fully in-person team in Cupertino, and greatly value engineering skills. We do not have boundaries between engineering and research, and we expect all of our technical staff to contribute to both as needed.

Equal employment opportunity

Etched is an Equal Employment Opportunity employer; we do not discriminate on the basis of any protected group status under any applicable law.