Forward Deployed Engineer

Darwin Recruitment; California; On-site; USD 140,000 - 210,000; Full time

Job summary

Darwin Recruitment seeks a Forward Deployed Engineer for Bay Area, CA. You will work directly with research and enterprise teams to deploy infrastructure software across Linux systems, schedulers, and Kubernetes in AI/HPC environments.

The role combines hands-on troubleshooting, deployment ownership, performance optimisation, and feedback to product teams. Strong Linux and SLURM experience are essential.

Qualifications

• Around 5+ years of software engineering experience in production workload environments.
• Experience leading multi-month enterprise or research deployments end to end.
• Hands-on expertise deploying and administering SLURM with GPU resources.
• Strong Linux sysadmin knowledge: systemd, cgroups v2, namespaces, networking.

Responsibilities

• Lead customer integrations and deploy software into HPC environments.
• Configure and troubleshoot Linux, schedulers, Kubernetes, networking, and monitoring.
• Identify gaps and provide actionable feedback to product teams.
• Measure performance and design automation to improve workload handling.
• Own deployments and troubleshoot production issues across the stack.
• Develop installation guides and repeatable deployment processes.

Skills

Linux systems

Kubernetes

SLURM

Troubleshooting

Performance tuning

Tools

SLURM

Kubernetes

strace

dmesg

Job description

Title: Forward Deployed Engineer

Location: Bay Area, California

Salary: Competitive base salary

Industry: AI infrastructure, high-performance computing, and cloud systems

Overview of the Role:

We are working with an early-stage technology business developing infrastructure software that helps organisations improve the reliability, utilisation, and performance of AI and high-performance computing (HPC) environments. Its technology supports the movement and recovery of GPU workloads.

They are looking to hire a Forward Deployed Engineer. You will work directly with research, enterprise, and AI infrastructure teams to understand their environments, resolve technical challenges, and implement solutions across Linux systems, workload schedulers, Kubernetes, networking, and observability.

This is a hands-on, customer-facing engineering role combining systems-level troubleshooting, deployment ownership, performance optimisation, and feedback to product and engineering teams.

Responsibilities:

• Lead customer integrations and deploy software into HPC scheduling and container-orchestration environments.
• Configure and troubleshoot systems across Linux, workload schedulers, Kubernetes, node configuration, networking, and monitoring.
• Identify technical gaps in real-world deployments and communicate actionable feedback to product and engineering teams.
• Measure reliability, throughput, and platform performance; design policy-based automation to improve workload handling.
• Own critical deployments, investigate installation and production issues across the stack, and elevate issues where appropriate.
• Develop and maintain installation guides and repeatable deployment processes to accelerate onboarding for future customers.

Essential Requirements:

• Around 5+ years of software engineering experience, including configuring and managing production workload-scheduling environments.
• Experience leading a multi-month enterprise or research deployment end to end, from scoping through sign-off.
• Hands‑on experience deploying and administering SLURM, including controller and database services, accounting, cgroup integration, and GPU resource selection.
• Strong Linux systems knowledge, including systemd, cgroups v2, namespaces, networking, filesystems, firewalls, kernel-module loading, and PAM session modules.
• Ability to investigate low-level system behaviour using tools such as strace and dmesg and develop evidence‑based troubleshooting hypotheses.
• Operational experience with Kubernetes, including operators, custom resource definitions, networking interfaces, device plugins, and node-level debugging.

Beneficial Experience:

• Experience working in a field engineering team for an HPC integrator.
• Familiarity with CRIU, container runtimes, GPU driver internals, or distributed‑training systems.
• Hands‑on experience with AI workload orchestration tools such as NVIDIA Dynamo, Determined, Ray, Kueue, KServe, or comparable platforms.
• Contributions to open‑source workload schedulers or job‑management systems, such as SLURM, Flux, Torque, or PBS.