Senior Software Engineer, LLM Platform

FDE Times; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

Equity

401(k) Match

Paid Parental Leave

Fertility Benefits

Medical Insurance

Dental Insurance

Vision Insurance

Mental Health Support

Wellness Stipend

Learning Stipend

Internet Allowance

Commuting Benefits

Free Lunch

Gym Access

Flexible Paid Time Off

Paid Holidays

Team Outings

Referral Bonuses

Job summary

FDE Times is seeking a senior software engineer to build and operate scalable LLM infrastructure in a multi-tenant environment. You will work on LiteLLM gateways, provider integrations, routing, access controls, and observability, with on-call responsibilities to ensure reliability.

You will collaborate with engineering, data science, and product teams to improve developer enablement and platform robustness while focusing on scalability, cost attribution, and capacity planning.

Qualifications

• Strong production software engineering experience.
• Hands-on knowledge of high-throughput gateways or multi-tenant services.
• Practical experience with LLM APIs, access/credential management, distributed observability, incident response, and usage metering/capacity planning.

Responsibilities

• Build and operate shared LLM infrastructure and gateways.
• Partner with engineering, data science, product, and FDE teams.
• Provide on-call support and improve reliability and developer enablement.

Skills

Kubernetes

Terraform

CI/CD

LLM Platform Engineering

API Gateways

Observability

Incident Response

Cost Attribution

Capacity Planning

Model Serving

Workflow Orchestration

Python

TypeScript

Go

LLM Provider APIs

Authentication and Access Controls

Tools

Job description

Build and operate shared LLM infrastructure, including LiteLLM gateways, provider integrations, routing, access controls, observability, cost attribution, and hosted or self-hosted inference. Improve reliability and developer enablement across the broader data and ML platform, partnering with engineering, data science, product, and FDE teams while participating in on‑call support.

Requirements: Candidates should have strong production software engineering experience and experience operating high-throughput gateways or multi-tenant services, with hands‑on knowledge of Kubernetes, Terraform, CI/CD, and production service ownership. They should also have practical experience with LLM APIs, access and credential management, distributed‑systems observability, incident response, and usage metering or capacity planning.

Key Skills: Python, TypeScript, Go, LLM Platform Engineering, API Gateways, Kubernetes, Terraform, CI/CD, LLM Provider APIs, Authentication and Access Controls, Observability, Incident Response, Cost Attribution, Capacity Planning, Model Serving, Workflow Orchestration

Benefits: Equity, 401(k) Match, Paid Parental Leave, Fertility Benefits, Medical Insurance, Dental Insurance, Vision Insurance, Mental Health Support, Wellness Stipend, Learning Stipend, Internet Allowance, Commuting Benefits, Free Lunch, Gym Access, Flexible Paid Time Off, Paid Holidays, Team Outings, Referral Bonuses