Senior Software Engineer — AI Systems & Scalability

Boodlebox

United States

Remote

USD 175,000 - 185,000

Full time

7 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Unlimited PTO
Medical, dental, and vision coverage
401(k)
Fully remote work environment

Job summary

BoodleBox is seeking a Senior Software Engineer for AI Systems & Scalability. You will own production backends in Python and Scala, optimize latency and throughput, and explore self-hosted inference to reduce costs.

You’ll operate in a fully remote environment across the United States, contributing to a GenAI platform used by 100,000+ users. You will focus on designing scalable services, monitoring, and cost-aware architectures while collaborating with teams to deploy robust AI capabilities.

Qualifications

  • Proficient in production backend coding in Python and Scala.
  • Experience diagnosing distributed systems under load.
  • Experience serving a model behind an API with concurrency controls.

Responsibilities

  • Production services in Scala and Python with robust monitoring.
  • Improve scalability and reduce latency under load.
  • Evaluate and implement self-hosted inference architectures.
  • Maintain practical MLOps for production models without training pipelines.
  • Own deployed AI services with clear ownership and backups.
  • Track load, cost per request, and latency metrics before/after optimizations.

Skills

Python
Scala
MLOps
Distributed systems
API exposure

Tools

Kubernetes

Job description

Senior Software Engineer, AI Systems & Scalability

Full-time Remote, United States

Make BoodleBox’s AI platform faster and cheaper per request across Scala and Python, and lead the move to self-hosted inference where the numbers support it.

About BoodleBox

BoodleBox is a VC-backed GenAI platform built for higher education. With 100,000+ users across 1,200+ colleges and universities, we partner with faculty, staff, and students to make responsible AI adoption practical, impactful, and lasting. We're guided by three core values: service, education, and collaboration.

The Role

You're a backend engineer who is at home in a Scala service and a Python service on the same day. You take it personally when a request is slow. You've stood up a model behind an API under real load, watched the GPU utilization, and can tell us what it cost per request. You have enough MLOps to run models in production and no interest in training them. You find the bottleneck before you ask for more machines.

BoodleBox is already a generative AI platform in production: Scala and Python microservices on Kubernetes, hundreds of thousands of users, models served today through managed providers. We want to make the one we have faster, more scalable, and more flexible about where models run — including evaluating and implementing self-hosted inference for embeddings, re-rankers, document-processing models, and possibly open-weight LLMs.

What You'll Own
  • Production services in Scala and Python. The integrations, middleware, and services that connect generative AI to the application platform, built with monitoring and recovery from the start.
  • Scalability and latency. Finding and removing bottlenecks in request handling, throughput, and concurrency so the platform handles more users at lower cost per request.
  • Self-hosted inference. The evaluation, the serving architecture, GPU utilization, autoscaling, and the honest cost comparison against managed providers.
  • Practical MLOps. Models monitored in production, inference infrastructure versioned and operated reliably. No training pipelines.
  • Your services in production. A set of AI services you steward with a named backup: healthy, documented, safe for other engineers to change.
  • The numbers. Load, cost per request, p99 latency — before and after, written down.
Who You Are
  • You write production backend code in Python and Scala. Strong in Python, and either strong in Scala or demonstrably quick to get there on the JVM.
  • You've diagnosed a distributed system under load. From logs, metrics, and traces, to the actual cause, not to "add replicas."
  • You've served a model behind an API. Concurrency, resource limits, the moment it fell over, what you did.
  • You know the managed-versus-self-hosted trade. When it's worth running it yourself and when it isn't, in dollars and in pages.
  • You own outcomes. Design through deploy through the 3am follow-through.
  • You're comfortable being the specialist, not the lead. Someone else is growing into leading this group. You're here for depth.
What We Offer
  • Competitive base salary of $175,000–$185,000, plus bonus and equity.
  • Unlimited PTO.
  • Medical, dental, and vision coverage. BoodleBox covers 75% of premiums.
  • 401(k).
  • Fully remote work environment.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Systems Engineer — Retrieval & Context
Senior AI Systems Engineer — Retrieval & Context

Boodlebox • Colorado Springs (CO)

Hybrid
USD 175,000 - 185,000
Competitive base salary
Unlimited PTO
Healthcare coverage
+3
Deal Desk & Contracts Specialist
Deal Desk & Contracts Specialist

BoodleBox • United States

Remote
USD 105,000 - 120,000
Unlimited PTO
Medical coverage
Dental coverage
+4
Software Engineer - Model Performance Systems
Software Engineer - Model Performance Systems

Baseten • Seattle (WA)

On-site
USD 165,000 - 330,000
Competitive compensation & equity
US medical/dental/vision insurance for
flexible PTO including Winter Break
+3
Applied AI/ML Engineer — Build AI at Scale (Remote)
Applied AI/ML Engineer — Build AI at Scale (Remote)

Boundless Networks Ltd. • United States

Remote
USD 175,000 - 250,000
Equity allocation
Health, dental, vision
Flexible PTO
+3
Applied AI/ML Engineer
Applied AI/ML Engineer

Boundless Network • United States

On-site
USD 175,000 - 250,000
Equity
Health/dental/vision
Flexible PTO
+2
Senior Software Engineer (AI Infra + Systems)
Senior Software Engineer (AI Infra + Systems)

Jobot • United States

Remote
USD 170,000 - 200,000
Fully remote
Competitive compensation
Lucrative Equity
+1
Senior Software Engineer (AI Infra + Systems)
Senior Software Engineer (AI Infra + Systems)

Jobot • Atlanta (GA)

Remote
USD 170,000 - 200,000
Fully Remote
Equity
Healthcare benefits
+1
Remote AI/ML Engineer — Production Inference & RL
Remote AI/ML Engineer — Production Inference & RL

Boundless • United States

Hybrid
USD 175,000 - 250,000
Competitive salary + equity
Health/dental/vision
Flexible PTO
+2
Senior Native Experience Engineer — iOS & Android
Senior Native Experience Engineer — iOS & Android

Boodlebox • United States

Remote
USD 175,000 - 185,000
Unlimited PTO
Medical, dental, and vision coverage
75% premiums covered by BoodleBox
+4
Software Engineer, Developer Productivity - Inference Runtime
Software Engineer, Developer Productivity - Inference Runtime

OpenAI, Inc. • San Francisco (CA)

On-site
USD 230,000 - 385,000