Staff AI Product Engineer

nscaleoperationsukltd

Georgia

On-site

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Nscale is seeking a Staff AI Engineer (Specialized) to set technical direction for inference and RL at the core of its AI services platform. You will own architecture for how models are served on Nscale's GPU cloud and expose clean interfaces to customers and internal teams.

This role drives system-wide performance, cost efficiency, and standards across 2–4 teams covering serving, post-training, and platform, while mentoring engineers and collaborating with research, product, and infrastructure

Qualifications

  • 8–12 years of engineering experience in production AI systems at scale.
  • Ability to set technical direction for an AI systems domain.
  • Deep expertise in production LLM inference and memory management.
  • Hands-on RL for LLMs including RLHF and reward modelling.
  • Experience designing developer-facing APIs and SDKs.
  • Cross-team influence and standards adoption.
  • Experience AI infrastructure with clear control/data plane separation.

Responsibilities

  • Define inference serving architecture including routing, batching, and cache management.
  • Lead RL and post-training systems design, RLHF and related methods.
  • Set standards for developer APIs, SDKs, OpenAPI specs and tooling.
  • Coach junior engineers and align with research, product, and infrastructure leadership.

Skills

AI systems leadership
LLM inference
RL for LLMs
API design

Tools

OpenAI-compatible APIs

Job description

About Nscale

Nscale is taking on the hyperscalers by building a vertically integrated GenAI cloud platform. We own the data centers, software, and applications that power today's AI stack using sustainable technology solutions. We thrive on a culture of relentless innovation, ownership, and accountability, where every team member takes pride in their work and drives it with excellence and urgency. As a Nscaler, you'll build trust through openness and transparency, where everyone is inspired to do their best work. Collaboration is key, and we work together swiftly and respectfully, embracing adaptability and resilience in all we do.

About the Role

Nscale is looking for a Staff AI Engineer (Specialized) to set technical direction for the inference and reinforcement learning systems at the core of our AI services platform - and for the APIs through which other engineers consume them.

This role owns the architecture of how models are served on Nscale's GPU cloud, how RL and post-training workloads run on it, and how both are exposed to customers and internal teams as clean, reliable, high-performance interfaces. You'll work across 2-4 teams spanning serving, post-training, and platform, defining how these systems are built and establishing the standards that create engineering leverage across the organization.

As a Staff engineer, you are the technical authority for this area of the AI stack. Your decisions determine the latency, throughput, and cost profile of every token Nscale serves, and the correctness and efficiency of every RL run on our platform. You resolve ambiguous architectural questions where the answer space is genuinely open - disaggregated versus co-located serving, on-policy versus off-policy RL infrastructure, where the API boundary should sit - and your solutions become the standards others build on.

Responsibilities
Inference
  • Set technical direction for Nscale's inference serving architecture: request routing, scheduling, continuous batching, KV cache management, prefix caching, and speculative decoding
  • Drive the strategy for model efficiency in production - quantization (FP8, INT8/4), sparsity, pruning, distillation, and MoE serving - and the trade-offs between cost, latency, throughput, and model quality
  • Lead the resolution of systemic performance and reliability challenges across the serving stack, from kernel-level bottlenecks to fleet-level capacity and multi-tenant isolation
Reinforcement learning & post-training
  • Own the architecture of Nscale's RL and post-training systems: RLHF, DPO/GRPO-style methods, reward modelling, and agentic RL with tool calling, off-policy training, and decoupled sampling and policy updates
  • Define how inference and training share infrastructure in RL loops - rollout generation, sample buffering, weight synchronization, and the serving engine's role inside the training system
  • Establish standards for fine-tuning services (LoRA, QLoRA, adapters, full fine-tuning) and the data curation and processing workflows that feed them
Developer APIs & platform
  • Set the design standards for Nscale's developer-facing APIs, SDKs, and tooling - OpenAI-compatible and native interfaces, OpenAPI 3.x specifications, versioning, and rate limiting - so that inference and RL capabilities are consumable by engineers who never see the underlying systems
  • Create reusable frameworks and tooling that multiply the effectiveness of other AI engineers across Nscale
Leadership & direction
  • Coach and grow more junior engineers across teams; raise technical capability and engineering quality broadly
  • Collaborate with research, product, and infrastructure leadership to align inference and RL platform strategy with customer demand and business direction
  • Evaluate emerging serving engines, RL frameworks, and accelerator technologies; make decisive build/adopt/contribute recommendations
  • Represent the inference and RL domain in cross-organizational architecture reviews and strategic planning
Requirements
  • 8-12 years of engineering experience, with significant depth in production AI systems at scale (AI labs, hyperscalers, or leading ML infrastructure companies)
  • Demonstrated ability to set technical direction for an AI systems domain at scale
  • Deep expertise in production LLM inference: serving architectures, KV cache and memory management, batching and scheduling strategies, speculative decoding, and low-precision inference
  • Strong hands‑on expertise in RL for LLMs - RLHF, DPO and related preference-optimization methods, reward modelling, or agentic/multi-turn RL - including the systems that make them run efficiently on GPU clusters
  • Proven ability to design developer-facing APIs and SDKs that are clean, versioned, and adopted by other engineers and external customers
  • Strong cross-team influence: track record of creating standards and practices adopted by multiple teams
  • Experience designing AI infrastructure with clear control plane / data plane separation and cell-based archite
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal AI Product Engineer
Principal AI Product Engineer

nscaleoperationsukltd • Georgia

On-site
USD 260,000 - 360,000
Staff AI Product Engineer
Staff AI Product Engineer

Nscale • Seattle (WA)

On-site
USD 220,000 - 293,000
Principal AI Product Engineer
Principal AI Product Engineer

Nscale • Houston (TX), New York (NY), San Francisco (CA), Seattle (WA)

On-site
USD 290,000 - 520,000
Medical insurance
Dental insurance
Vision insurance
+3
Staff AI Product Engineer
Staff AI Product Engineer

Nscale • New York (NY)

On-site
USD 220,000 - 330,000
Staff AI Platform Architect: Inference & RL
Staff AI Platform Architect: Inference & RL

nscaleoperationsukltd • Georgia

On-site
USD 180,000 - 240,000
AI Infrastructure Engineer
AI Infrastructure Engineer

Pokee AI • United States

Remote
USD 140,000 - 220,000
Staff AI Platform Engineer – Inference & RL Architect
Staff AI Platform Engineer – Inference & RL Architect

Nscale • Seattle (WA)

On-site
USD 220,000 - 293,000
Principal AI Systems Architect - Inference & Post-Training
Principal AI Systems Architect - Inference & Post-Training

Nscale • Houston (TX), New York (NY), San Francisco (CA), Seattle (WA)

On-site
USD 290,000 - 520,000
Medical insurance
Dental insurance
Vision insurance
+3
Inference Infrastructure Architect (Remote)
Inference Infrastructure Architect (Remote)

telnyx54 • United States

Remote
USD 180,000 - 320,000
Founding-seat influence on platform
Remote position
Open-source contribution & conference"
Member of Technical Staff: Core AI Platform
Member of Technical Staff: Core AI Platform

Lightspeed AI, Inc. • Seattle (WA)

Hybrid
USD 180,000 - 240,000