AI Inference Engineer (GPU/Rust/CUDA)

Perplexity; New York (NY); On-site; USD 220,000 - 485,000; Full time

Job summary

Perplexity in New York is seeking an experienced engineer to join our team and own ML inference workloads, building a Rust/Python-based serving runtime and CUDA kernel work that scales to production. You will work with transformer-based models, enabling efficient weight loading, scheduling, and KV-cache management in our in-house infrastructure.

You will read research papers, implement kernels, and diagnose production incidents in a fast-moving environment, collaborating across languages and

Qualifications

• 3+ years of professional software engineering experience with meaningful work on ML inference or high-performance systems.
• Familiarity with at least one deep learning framework (PyTorch, JAX, TensorFlow).
• Understanding of GPU architectures (memory hierarchy, warp scheduling, tensor cores).
• Understanding of common LLM architectures and inference optimization techniques (e.g. quantization, speculative decoding, prefill-decode disaggregation).

Responsibilities

• New models support for transformer-based retrieval, text-generation, and multimodal models in the inference infrastructure.
• Port CUDA kernels to CuTe DSL and migrate GPU kernels for portability.
• Develop a Rust-native serving runtime to handle growing traffic and reduce Python pains.
• Profile and optimize performance across network, batching, and GPU kernel interleaving.
• Build dashboards, alerts, and automated remediation to catch regressions.

Skills

ML inference

High-performance systems

Distributed systems

Rust

Python

CUDA

Tools

PyTorch

JAX

TensorFlow

Job description

Perplexity in New York is seeking an experienced engineer to join our team and own ML inference workloads, building a Rust/Python-based serving runtime and CUDA kernel work that scales to production. You will work with transformer-based models, enabling efficient weight loading, scheduling, and KV-cache management in our in-house infrastructure.

You will read research papers, implement kernels, and diagnose production incidents in a fast-moving environment, collaborating across languages and