Senior Software Engineer (AI Inference & Runtime Platform)

Jobgether; United States; Remote; USD 140,000 - 230,000; Full time

Benefits offered by this job

Salary

Equity

Bonus

Healthcare

Flexible time off

Remote work

Company summits

Open-source impact

Job summary

Jobgether is partnering with a US‑based team seeking a Senior Software Engineer (AI Inference & Runtime Platform). You will own scalable AI infrastructure, from high‑performance model inference to secure sandbox execution, across Rust, Python and cloud platforms.

Expect to work on Kubernetes clusters, GPU resources and open‑source contributions. You should have 5+ years in production systems, strong Rust or willingness to use Rust, and hands‑on experience with GPU orchestration, cryptographic

Qualifications

• At least 5 years delivering production systems with a systems programming language, preferably Rust.
• Experience operating Kubernetes workloads, controllers or operators, autoscaling, and node lifecycle management.
• Knowledge of secure AI‑agent execution, least‑privilege access, and audit trails.
• Experience deploying open‑weight LLM inference infrastructure and production endpoints.

Responsibilities

• Manage model-serving infra for open‑weight LLMs, including upgrades and canaries.
• Operate Kubernetes infra for inference and sandbox workloads with GPU scheduling.
• Own the data plane end‑to‑end with reliable recovery procedures.
• Develop secure agent sandbox runtime and host control plane.
• Build FastAPI control-plane services and IaC dashboards.

Skills

Rust

Go/C/C++/Zig

Kubernetes

Distributed systems

Python

Async runtimes

Open-source

Education

Bachelor's degree

Tools

Kubernetes

Terraform/OpenTofu

Bicep

OpenTelemetry

AWS/Azure/GCP

Job description

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Software Engineer (AI Inference & Runtime Platform) based in the United States.

Join a mission-driven engineering team building advanced AI infrastructure to address complex challenges across critical industries, including energy, sustainability, logistics, and climate technology. In this role, you will own the runtime foundations that power AI workloads, from high-performance model inference to secure execution environments for AI-generated code. You will design and operate scalable systems across Kubernetes clusters, GPU infrastructure, and cloud platforms. Working across Rust, Python, distributed systems, and infrastructure automation, you will help deliver reliable, cost-efficient, and secure AI services. This opportunity is ideal for an engineer who has built and operated production-grade inference or serverless GPU platforms and enjoys solving challenging systems problems. You will also contribute to open-source software and help establish engineering standards as the platform evolves.

Accountabilities

• Manage the model-serving infrastructure for open-weight large language models, including deployment, configuration, cold-start optimization, model-specific service-level objectives (SLOs), upgrades, and canary releases.
• Operate and improve the Kubernetes infrastructure supporting inference and sandbox workloads, including custom operators, custom resource definitions (CRDs), autoscaling, GPU scheduling and sharing, and node lifecycle management.
• Own the stateful data plane end to end, ensuring reliable operation, documented recovery procedures, and regularly tested restoration processes.
• Develop and maintain the secure agent-sandboxing runtime and host-side control plane, covering lifecycle management, command execution, snapshots, forking, teardown, usage metering, and isolation-boundary security.
• Build and maintain FastAPI control-plane services, infrastructure-as-code configurations using Terraform/OpenTofu and Bicep, and dashboards that support effective operations.
• Provide the infrastructure foundation used by backend engineering teams, collaborating across services and investigating issues that span the platform and application layers.
• Establish and maintain observability, performance, reliability, and operational practices that help teams diagnose problems and make informed decisions.
• Contribute to an open-source-oriented development process through reviewed pull requests, comprehensive documentation, reproducible builds, and releases as components mature.

Requirements

• At least 5 years of experience delivering production systems using a systems programming language. Strong Rust expertise is valuable, while deep experience in Go, C/C++, or Zig combined with a genuine willingness to work in Rust is also relevant.
• Solid understanding of asynchronous runtimes, memory safety, and low-level debugging, including troubleshooting at the system-call boundary.
• Hands‑on experience operating Kubernetes workloads that other teams depend on, including controllers or operators, scheduling, autoscaling, and node lifecycle management.
• Proven ability to threat‑model isolation boundaries involving technologies such as namespaces, cgroups, seccomp, and hypervisors, including assessing what an untrusted guest could observe, manipulate, or exhaust.
• Practical knowledge of secure AI‑agent execution, including least‑privilege access, preventing credentials from entering sandbox environments, maintaining audit trails, and requiring human approval for actions that modify systems of record.
• Experience deploying or operating open‑weight LLM inference infrastructure, such as vLLM, SGLang, or comparable serving systems, and packaging models into reliable, metered production endpoints.
• Strong distributed‑systems performance skills, including measurement‑driven optimization, latency analysis, and the ability to demonstrate improvements with concrete metrics.
• Practical experience with technologies such as Rust and Tokio, Python and FastAPI, Kubernetes operators, controller‑runtime, Kubebuilder, CRDs, KEDA, Karpenter, GPU device plugins, or Dynamic Resource Allocation (DRA), along with a willingness to learn unfamiliar components.
• Familiarity with isolation technologies such as Firecracker, Kata Containers, gVisor, or equivalent solutions.
• Experience with secrets management and egress controls, including Vault‑or‑KMS‑class systems, as well as stateful technologies such as pgvector, Qdrant, or Neo4j, with strong backup and failover practices.
• Comfort operating across AWS, Azure, GCP, and managed GPU cloud environments, using infrastructure‑as‑code tools such as Terraform/OpenTofu and Bicep, along with observability tooling such as OpenTelemetry.
• A bachelor's degree is required; a master's degree is a plus.
• Ability to travel twice a year for company summits.
• Current authorization to work full‑time in the United States. Employment visa sponsorship or transfer is not available.
• Willingness to complete a written take‑home assignment followed by a live, two‑hour technical session with the engineering team.
• Applicants may apply to a maximum of two roles at a time. Applying to more than two roles within a six‑month period results in automatic disqualification.

Benefits

• Salary: $140,000–$230,000, with compensation determined by experience, capabilities, and location.
• Equity: Equity participation in an early‑stage company.
• Bonus: Eligibility for performance‑related bonus compensation.
• Healthcare: Health insurance with meaningful coverage for dependents.
• Flexible time off: Paid time off designed to support flexibility and work‑life balance.
• Remote work: Fully remote working culture, with a cluster of teammates based in Seattle.
• Mission‑driven work: The opportunity to contribute to AI transformation across critical industries through impactful applications in energy, sustainability, climate, logistics, and related fields.
• Engineering ownership: The opportunity to shape foundational AI infrastructure, work across the stack, and contribute to open‑source projects.
• Company summits: Opportunities to connect with teammates in person twice per year.

How Jobgether Works

We use an AI‑powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top‑fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre‑contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.