Inference Systems Performance Architect

Socket.dev; San Jose (CA); On-site; USD 245,000 - 325,000; Full time

Benefits offered by this job

Health Insurance

Dental Insurance

Vision Insurance

Disability Insurance

Life Insurance

AD&D Insurance

HSA with employer contribution

Headspace

Gympass+

One Medical

Employee Assistance Program

Job summary

SambaNova Systems is seeking an Architect on the Inference Systems Performance team to own end-to-end performance for large-scale LLM inference, from tokenization to deployment, across hardware and software components. You will drive reproducible workload capture, benchmarking, modeling, and simulation to inform capacity planning against customer SLOs.

Lead the performance-modeling effort, mentor senior engineers, and collaborate with model optimization, systems, and product teams to balance

Qualifications

• 12+ years of experience in performance engineering, with a demonstrated record of technical leadership on large-scale, complex systems.
• Deep expertise in end-to-end performance analysis of distributed systems with many moving parts and the ability to localize bottlenecks that others cannot.
• Proven command of realistic workload generation and simulation and of performance modeling, including calibrating models against real, variable workloads.
• Demonstrated ability to enter an unfamiliar domain and apply core performance methods with transferable discipline expertise.
• Ability to lead cross-functional efforts, mentor senior engineers, and influence organizational direction.
• Experience representing an organization's credibly to customers and partners
• Track record of independently scoping and delivering high-complexity, high-ambiguity work with significant impact on products or roadmap

Responsibilities

• Define and drive the technical strategy for inference-systems performance including workload capture, benchmarking, modeling, and simulation, while developing an architecture that enables many potential futures
• Build the workload-capture and agentic-benchmarking capability - capture representative production traffic and enforce the discipline of interrogating results
• Own the performance-modeling and simulation practice - models that predict how a configuration change moves the output, informing capacity planning against customer SLOs and next-generation system and hardware planning
• Attack the end-to-end profiling gap - drive tooling that produces accurate, actionable profiles of a distributed inference pipeline so bottlenecks can be localized across host, accelerator, and fabric
• Serve as the senior technical voice across model-optimization, systems, hardware, and product, tying together multiple engineering activities and teams, and weighing trade-offs of reliability, scalability, operational cost, and ease of adoption
• Act as a resource for the entire organization including representing SambaNova's performance story to customers and partners
• Mentor and multiply by raising the capability of principal and senior engineers, building the systems, tools, and patterns that make everyone more productive
• Drive the resolution of the most ambiguous, novel challenges that span organizational boundaries or have no established answer in the field yet

Skills

Performance engineering

Distributed systems

Workload generation

Performance modeling

Leadership

Cross-functional collaboration

Customer storytelling

Tools

LLM inference serving frameworks

Job description

The era of pervasive AI has arrived. In this era, organizations will use generative AI to unlock hidden value in their data, accelerate processes, reduce costs, drive efficiency and innovation to fundamentally transform their businesses and operations at scale. SambaNova Suite™ is the first full-stack, generative AI platform, from chip to model, optimized for enterprise and government organizations. Powered by the intelligent SN40L chip, the SambaNova Suite is a fully integrated platform, delivered on-premises or in the cloud, combined with state-of-the-art open-source models that can be easily and securely fine-tuned using customer data for greater accuracy. Once adapted with customer data, customers retain model ownership in perpetuity, so they can turn generative AI into one of their most valuable assets.

About the role

As an Architect on the Inference Systems Performance team, you'll own the discipline of end-to-end performance for large-scale LLM inference at SambaNova, from how a request moves through tokenization, prefill, decode, and the fabric between them, to how an entire deployment is sized against customer SLOs. Inference-systems performance is a nascent field; the results of design choices are being discovered daily rather than inherited from a mature craft, and this role exists to bring rigor to that frontier. The work spans two coupled pillars. The first is reproducible workload capture and benchmarking -- building faithful, replayable representations of real and increasingly agentic traffic, so that what we measure reflects production rather than an artifact of a naive load script. The second is performance modeling and simulation - analytic and simulation models that turn measurement into a "what-if" capability, letting us reason about configurations and hardware that do not exist yet. Together these feed both today's serving optimization and the next generation of system planning. The technical frontier you'll help define is heterogeneous, disaggregated inference - GPU on prefill, the RDU on decode - which explores hard problems across networking, storage, prompt caching, and tail-latency-bound data movement. You will be the go-to person for inference-systems performance across SambaNova, and a resource the entire organization relies on to answer "how fast can this go, and what will it take."

Responsibilities

• Define and drive the technical strategy for inference-systems performance including workload capture, benchmarking, modeling, and simulation, while developing and architecture that enables many potential futures
• Build the workload-capture and agentic-benchmarking capability - capture representative production traffic and enforce the discipline of interrogating results, spotting artificial contention or misleadingly high cache-hit rates that never occur in real use
• Own the performance-modeling and simulation practice - models that predict how a configuration change moves the output, informing capacity planning against customer SLOs and next-generation system and hardware planning
• Attack the end-to-end profiling gap - drive tooling that produces accurate, actionable profiles of a distributed inference pipeline so bottlenecks can be localized across host, accelerator, and fabric
• Serve as the senior technical voice across model-optimization, systems, hardware, and product, tying together multiple engineering activities and teams, and weighing trade-offs of reliability, scalability, operational cost, and ease of adoption
• Act as a resource for the entire organization including representing SambaNova's performance story to customers and partners
• Mentor and multiply by raising the capability of principal and senior engineers, building the systems, tools, and patterns that make everyone more productive
• Drive the resolution of the most ambiguous, novel challenges that span organizational boundaries or have no established answer in the field yet

Required qualifications

• 12+ years of experience in performance engineering, with a demonstrated record of technical leadership on large-scale, complex systems
• Deep expertise in end-to-end performance analysis of distributed systems with many moving parts and the ability to localize bottlenecks that others cannot
• Proven command of realistic workload generation and simulation and of performance modeling, including calibrating models against real, variable workloads
• Demonstrated ability to enter an unfamiliar domain and apply core performance methods with transferable discipline expertise
• Ability to lead cross-functional efforts, mentor senior engineers, and influence organizational direction
• Experience representing an organization's credibly to customers and partners
• Track record of independently scoping and delivering high-complexity, high-ambiguity work with significant impact on products or roadmap

Preferred qualifications

• Direct experience with LLM inference serving - continuous batching, prompt/KV caching, prefill/decode disaggregation, tail-latency SLOs
• Familiarity with inference simulation frameworks or agentic benchmarking efforts
• A public technical voice - talks, writing, or community presence on systems performance

Base Salary Range: Base Pay Range $245,000 - $325,000 USD

EEO Policy

SambaNova Systems is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard basis of age (40 and over), color, disability, gender identity, genetic information, marital status, military or veteran status, national origin/ancestry, race, religion, creed, sex (including pregnancy, childbirth, breastfeeding), sexual orientation, and any other applicable status protected by federal, state, or local laws.

Benefits Summary for US-Based, Full-Time Employment Positions

• We cover 95% premium coverage for employee medical insurance, and 77% premium coverage for dependents and offer a Health Savings Account (HSA) with employer contribution.
• We also offer Dental, Vision, Short/Long term Disability, Basic Life, Voluntary Life, and AD&D insurance plans in addition to Flexible Spending Account (FSA) options like Health Care, Limited Purpose, and Dependent Care.
• Our library of well-being benefits available to you and your dependents includes a full subscription to Headspace, Gympass+ membership with access to physical gyms, One Medical membership, counseling services with an Employee Assistance Program, and much more.