AI Inference Platform Engineer

BaseTen; New York, San Francisco (NY, CA); On-site; USD 140,000 - 210,000; Full time

Benefits offered by this job

Equity

Medical, dental and vision insurance (

Flexible PTO including Winter Break

Parental leave

Fertility and family-building stipend

Company 401(k)

Learning and networking opportunities

Job summary

Baseten is hiring for a Model Performance Engineer focused on fast, reliable model APIs and multi-GPU serving. You will optimize inference runtimes, build scalable backends, and improve developer experiences across the platform.

You will work with TensorRT, CUDA tracing, and advanced scheduling to push performance boundaries while collaborating with product and infra teams. This role is well-suited for engineers who love practical systems at scale.

Qualifications

• 3+ years experience building and operating distributed systems or large-scale APIs.
• Proven track record of owning low-latency, reliable backend services (rate-limiting, auth, quotas, metering, migrations).
• Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management.
• Comfortable debugging complex systems, from runtime internals to GPU execution traces.
• Strong written communication; able to produce clear design docs and collaborate across functions.

Responsibilities

• Design, build, and operate the Model APIs surface with focus on advanced inference capabilities: structured outputs, tool/function calling and multi-modal serving.
• Profile and optimize TensorRT-LLM kernels, analyze CUDA, implement custom operators, and optimize memory and GPU throughput.
• Productionize performance improvements across runtimes with speculative decoding, quantization, batching, and KV-cache reuse.
• Instrument observability (metrics, traces, logs) and build benchmarks to measure speed, reliability, and quality.
• Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication.
• Collaborate with other teams to deliver robust, developer-friendly model serving experiences.

Skills

Distributed systems

Backend services

Low latency

Profiling & debugging

Technical communication

Tools

Kubernetes

API gateways

Service meshes

Distributed scheduling

Job description

Baseten is hiring for a Model Performance Engineer focused on fast, reliable model APIs and multi-GPU serving. You will optimize inference runtimes, build scalable backends, and improve developer experiences across the platform.

You will work with TensorRT, CUDA tracing, and advanced scheduling to push performance boundaries while collaborating with product and infra teams. This role is well-suited for engineers who love practical systems at scale.