Sr. Staff Inference Serving Engineer

Groq, Inc.; United States; Hybrid; USD 341,000 - 402,000; Full time

Job summary

Groq, Inc. is seeking a seasoned engineer to optimize an inference serving stack for fast, scalable product delivery. You will work on a system spanning trained models to high-throughput token output, requiring deep model internals, runtime software, and orchestration across GPUs and accelerators.

The role emphasizes performance benchmarks, batching, memory management, quantization, and cross-team collaboration with infrastructure and vendors. Remote work flexibility exists while offices open.

Qualifications

• BS/MS/PhD in CS, CE, EE or equivalent.
• 5+ years shipping performance-critical products.
• Strong understanding of transformer architectures.
• Fluency in at least one systems language and one high-level language.
• Measurement-driven approach to performance work.
• Clear communication about tradeoffs to stakeholders.

Responsibilities

• Build and optimize serving systems that handle high request volumes with low, predictable latency.
• Apply model compilation and graph optimization techniques, and benchmark to use them only where they add value.
• Implement and tune batching, caching, scheduling, and memory management for large models.
• Apply reduced-precision and quantization methods.
• Distribute models across multiple accelerators and nodes, balancing throughput and latency.
• Profile end-to-end performance, identify bottlenecks, and fix them—from kernels to routing, using lab and production telemetry.
• Partner with infrastructure and FDE teams to bring models into production quickly.
• Partner with vendors along the inference serving path to optimize the stack.

Skills

5+ years shipping performance-critical

Transformer architectures

Systems language + high-level language

Measurement-driven performance

Clear stakeholder communication

Education

BS / MS / PhD in CS, CE, EE or equivalent

Job description

Mission

GroqCore is building a high-performance software stack that converts bare metal compute into a token-producing engine This role is for a seasoned engineer with practical experience making LLMs fast, efficient, and reliable in production. This individual will work on an inference serving software stack from a trained model to a high-throughput delivery of output tokens. This will require experience on understanding/working with model internals, runtime software, and the supporting GPU & accelerator hardware underpinning it all.

Mission

GroqCore is building a high-performance software stack that converts bare metal compute into a token-producing engine This role is for a seasoned engineer with practical experience making LLMs fast, efficient, and reliable in production. This individual will work on an inference serving software stack from a trained model to a high-throughput delivery of output tokens. This will require experience on understanding/working with model internals, runtime software, and the supporting GPU & accelerator hardware underpinning it all.

Location

This role will be based in one of our three hiring hubs: the Dallas, San Francisco, or New York City area. The person hired for this role must be based in one of these three areas. You’ll have the flexibility to work remotely while we establish our local Groq office, with the expectation that this role will transition to onsite once the office opens.

Responsibilities & Opportunities In This Role

• Build and optimize serving systems that handle high request volumes with low, predictable latency
• Apply model compilation and graph optimization techniques, and benchmark carefully to use them only where they deliver real gains
• Implement and tune batching, caching, scheduling, and memory management strategies for large models
• Apply reduced-precision and quantization methods
• Distribute models across multiple accelerators and nodes, balancing throughput & latency
• Profile end-to-end performance, identify bottlenecks, and fix them—ranging from low‑level kernels to request routing—leveraging both lab/development environments and live telemetry from production systems.
• Partner with infrastructure and FDE teams to bring new models into production quickly
• Partner with vendors along the inference serving path in support of optimizing the stack

Ideal Candidates Have/are

• BS / MS / PhD in CS, CE, EE, or equivalent depth from industry
• 5+ years shipping performance‑critical products, with some portion of that serving models at scale
• A strong applied understanding of transformer architectures
• Fluency in at least one systems language and one high‑level language
• A rigorous, measurement‑driven approach to performance work
• Clear communication about tradeoffs to both technical and business stakeholders

Ways To Stand Out

• Experience writing or tuning custom kernels.
• Familiarity with non‑GPU or specialized inference hardware.
• Contributions to open‑source serving or compiler projects.

Why Join Us

• Purposeful Hiring: You’re not here by accident, and neither is anyone else. Every teammate is handpicked with intention because who we build with matters.
• Builders Wanted: You’re not just riding the rocket ship, you’re building it. Your work directly shapes the trajectory of our company.
• Mission‑Driven Work: We’re here to make a real impact. Our mission fuels everything we do.
• Tackling Hard Problems: If easy isn’t your thing, you’re in the right place. We solve some of the most complex and exciting challenges in our space.
• Excellence Is The Standard: High performance isn’t just encouraged, it’s the baseline. And it’s contagious.

Compensation

Groq is committed to providing competitive compensation through our Total Cash philosophy, which incorporates potential bonus value directly into base pay. The total cash salary range for this position, which is inclusive of the potential bonus value, is $341,400 - $401,600, with individual placement determined by your geographic location, experience, skills, and alignment with internal compensation standards. This range is specific to candidates located in the United States. Compensation for international candidates will vary based on local market dynamics. Beyond cash compensation, Groq also offers a Long‑Term Incentive (LTI) Program and a robust suite of employee benefits.