Senior Model API Engineer – Low-Latency AI Serving

Baseten; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

Baseten is seeking a backend-focused engineer for the Model Performance team to own low-latency APIs powering hosted model endpoints. You’ll work across distributed systems, model serving, and developer tooling to optimize throughput, reliability, and cost efficiency.

You’ll design and deploy benchmarking, profiling, and observability, implement platform fundamentals like authentication and quotas, and collaborate with multiple squads to deliver robust model serving experiences.

Qualifications

• 3+ years experience building and operating distributed systems or large‑scale APIs.
• Proven track record of owning low‑latency, reliable backend services (rate‑limiting, auth, quotas, metering).
• Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management.
• Strong written communication; able to produce clear design docs and collaborate across functions.

Responsibilities

• Design, build, and operate the Model APIs surface with focus on advanced inference capabilities and multi-GPU setups.
• Productionize performance improvements across runtimes and frameworks.
• Build benchmarking frameworks measuring real-world performance across model architectures and hardware.
• Instrument observability (metrics, traces, logs) and develop repeatable benchmarks for speed and reliability.
• Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication.
• Collaborate with other teams to deliver robust, developer-friendly model serving experiences.

Skills

Distributed systems

Large-scale APIs

Low-latency backend

Performance profiling

GPU performance

Technical writing

Job description

Baseten is seeking a backend-focused engineer for the Model Performance team to own low-latency APIs powering hosted model endpoints. You’ll work across distributed systems, model serving, and developer tooling to optimize throughput, reliability, and cost efficiency.

You’ll design and deploy benchmarking, profiling, and observability, implement platform fundamentals like authentication and quotas, and collaborate with multiple squads to deliver robust model serving experiences.