Member of Technical Staff, Inference Systems

Ashby, Inc.

Palo Alto (CA)

On-site

USD 180,000 - 280,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Competitive salary
Equity
Health coverage
401(k) match
Meals & discretionary budget

Job summary

Inco AI is hiring founding engineers to build the kernels, runtimes, and scheduling infrastructure for frontier models at scale. You will own latency, throughput, and cost per token across the serving stack, from kernels to inference models, with a focus on end-to-end improvement.

The role emphasizes hands-on work from first principles to production, with a preference for in-person collaboration in Palo Alto.

Qualifications

  • Experience optimizing latency- and throughput-critical inference across the full stack.
  • Familiarity with modern GPU systems and deployment architectures.
  • Working knowledge of serving infrastructure and inference stacks.

Responsibilities

  • Own end-to-end performance metrics: latency, throughput, cost per token.
  • Improve latency/throughput across kernels, runtimes, and serving layers.
  • Prototype to production across the serving stack for large-scale workloads.

Skills

Latency optimization
Throughput optimization
GPU systems
Kubernetes
Python
Rust/C++/Go

Tools

PyTorch
vLLM
TensorRT-LLM
Kubernetes
NVIDIA Dynamo
Kong
SMG

Job description

About the role

We are looking for founding members to build the systems that serve our frontier models at scale: the kernels, runtimes, batching, KV cache, load balancing, and scheduling that determine how quickly and cost-effectively each token is delivered across thousands of GPUs. As agentic workloads drive inference demand sharply higher, solving these problems will define the next generation of AI infrastructure.

Your mandate is to unlock the next order-of-magnitude gain in serving performance. You will work across the serving stack to improve latency, throughput, and cost per token, and when it is the highest-leverage path, go deeper into runtimes, kernels, quantization, or speculative decoding. We’re looking for builders who are energized by owning these outcomes end to end, rather than staying in one layer.

We built TensorRT-LLM and other tools that have shaped the field. As a founding engineer, you will help set the next standard. Our team is flat, the problems are consequential, and you will carry work from first principles and prototype to production serving.

About you
  • You chase performance wherever it lives, from kernels to serving systems to the model itself, and you think in terms of the whole system rather than a single layer

  • You would rather own a metric end to end (latency, throughput, cost per token) than a single component

  • You are drawn to the risk and upside of an early-stage startup

  • You embrace ambiguity and have a strong bias for action

  • You operate with a tech-lead mentality regardless of title or years of experience

  • You treat AI as a force multiplier: fluent with coding agents and modern AI/ML workflows, you fold them into how you ship, and you are willing to push the boundary of what’s possible with agents

Requirements
  • Experience optimizing latency- and throughput-critical inference across the full stack, from kernels and systems to the model layer

  • Familiarity with modern GPU systems and deployment architectures, including HGX B300 and GB300 NVL72

  • Familiarity with serving infrastructure, gateways, and inference stacks such as Kubernetes, NVIDIA Dynamo, Kong, SMG, PyTorch, vLLM, SGLang, and TensorRT-LLM

  • Working knowledge of modern LLM and multimodal architectures, including the quantization and speculative-decoding tradeoffs behind them

  • Production proficiency in Python and at least one systems language: Rust, C++, or Go

Bonus
  • Hands-on experience authoring kernels with CUDA, Triton, or PTX

  • Experience profiling, tracing, and debugging across model, runtime, kernel, and network layers using tools such as Nsight Systems (nsys)

  • Experience with batching, KV-cache management, and scheduling strategies

  • Knowledge of quantization formats and algorithms, including NVFP4, FP8, PTQ, QAT, and QAD

  • Familiarity with system-level inference optimization for NVL72-scale deployments, including wide expert parallelism, disaggregated prefill and decode, and related high-bandwidth interconnect techniques

  • A track record of running inference workloads across 1,000+ GPUs

  • A background in HPC or other high-performance systems

  • Contributions to open-source ML or systems infrastructure

Compensation & Benefits
  • Highly competitive salary and equity

  • Comprehensive health coverage for you and your dependents

  • Strong 401(k) match

  • Meals and snacks on us + discretionary productivity budget

Strong preference for in-person in Palo Alto or Brooklyn.

About Inco

Inco AI (inco.ai) builds the agent-native inference platform for tomorrow’s token economics. We build the inference layer that makes advanced AI fast enough for real workflows and affordable enough to use at scale. Agents are at the center of our work: when a single task chains dozens of model and tool calls, every gain in speed, cost, and reliability compounds.

We’re the researchers and engineers behind AWQ, DFlash, and TensorRT-LLM, with decades of combined leadership in AI efficiency and prior successful exits. Our team connects frontier research with the engineering needed to bring fast, reliable inference to production at scale.

We operate on five principles:

  1. Operate at the frontier. The field moves every day, and so do we. We reinvent ourselves constantly and push the frontier forward with our research. We keep process light and decisions fast, so a good idea reaches production in days, not quarters. Legacy doesn’t slow us down.

  2. Mission over individuals. We are obsessed with our customers and humble in pursuit of the goal. We share credit, name mistakes early, and change our minds when the evidence says so. The question is always what moves the mission forward, never who was right.

  3. Amplify with agents. Code is free now; architecture and process aren’t. We hand agents everything they can own, from code to operations, and spend our own time on design, judgment, and the problems only people can solve.

  4. Protect the maker’s time. Focus is our scarcest resource. We keep meetings few and protect long, uninterrupted stretches of work. We automate overhead, cover meals, and handle logistics, so our hours go to research and product.

  5. Be a team player. We win as one team. We share context generously, step in wherever we’re needed, and hold each other to a high bar, because exceptional work comes from exceptional peers.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff, Inference Research
Member of Technical Staff, Inference Research

Ashby, Inc. • Palo Alto (CA)

On-site
USD 200,000 - 350,000
Health coverage
401(k) match
Meals and snacks
+2
Member of Technical Staff, General
Member of Technical Staff, General

Ashby, Inc. • Palo Alto (CA)

On-site
USD 180,000 - 300,000
Competitive salary & equity
Health coverage for you and dependents
Strong 401(k) match
+1
Member of Technical Staff, Kernel Performance & Automation
Member of Technical Staff, Kernel Performance & Automation

Ashby, Inc. • Palo Alto (CA)

On-site
USD 180,000 - 280,000
Highly competitive salary and equity
Comprehensive health coverage for you/
Member of Technical Staff, Developer Infrastructure & Agent Automation
Member of Technical Staff, Developer Infrastructure & Agent Automation

Ashby, Inc. • Palo Alto (CA)

On-site
USD 180,000 - 240,000
Highly competitive salary and equity
Comprehensive health coverage for you
Strong 401(k) match
+1
Member of Technical Staff, Production
Member of Technical Staff, Production

Ashby, Inc. • Palo Alto (CA)

On-site
USD 180,000 - 280,000
Health coverage
401(k) match
Meals and snacks
+1
Member of Technical Staff, Developer Infrastructure & Agent Automation
Member of Technical Staff, Developer Infrastructure & Agent Automation

Inco AI • Palo Alto (CA)

On-site
USD 180,000 - 320,000
Salary & equity
Health coverage
401(k) match
+1
Inference Engineer
Inference Engineer

adaption • San Francisco (CA)

On-site
USD 150,000 - 230,000
Flexible work: In-person Bay Area coll
Adaption Passport: travel stipend
Lunch Stipend: weekly meal allowance
+1
Inference Engineer
Inference Engineer

adaption • United States

Hybrid
USD 180,000 - 280,000
Flexible work
Travel stipend
Lunch stipend
+1
Inference Engineer - Member of Technical Staff
Inference Engineer - Member of Technical Staff

Designworks Talent • Bellevue (WA)

Hybrid
USD 150,000 - 230,000
Medical, dental, and vision insurance
401(k) with company match
Paid holidays
Principal Engineer, Model Performance
Principal Engineer, Model Performance

Speedrun Talent Network • United States

Remote
USD 250,000 - 350,000
Stock options
Remote-first culture
Generous PTO
+2