Research Platform Engineer

Career Techniques Inc; New York (NY); Hybrid; USD 270,000 - 330,000; Full time

Job summary

Career Techniques seeks a senior platform engineer to build a multi-tenant research compute platform that orchestrates large-scale ML workloads across hybrid GPU/CPU infrastructures. You will create unified, elastic substrate enabling researchers to run simulations, distributed training, and complex pipelines with minimal overhead.

Responsibilities include architecting tenant isolation, multi-datacenter portability, and production-grade workflow orchestration, while delivering observability and

Qualifications

• Experience with HPC schedulers (e.g., Slurm) and containerized HPC execution.
• In-depth Kubernetes architecture and GPU-native schedulers.
• Experience with Ray, PyTorch, JAX in multi-node setups.

Responsibilities

• Elevate researcher experience by designing platform abstractions, APIs, and a durable job control plane.
• Architect multi-tenant scheduling and isolation across on-prem and cloud environments.
• Enable multi-datacenter portability and cloud bursting across on-premise datacenters and public cloud.
• Standardize workflow orchestration by evaluating and integrating production-grade engines.
• Build fault-tolerant pipelines with automated failure detection and retry mechanisms.
• Optimize data paths with high-speed networks and storage for large-scale workloads.

Skills

HPC schedulers

Kubernetes

Distributed ML

Workflow orchestration

Multi-platform infra

Storage & network

Observability

Leadership mindset

Tools

Ray

PyTorch

JAX

Job description

Summary

You will develop a multi-tenant research compute platform capable of dynamically orchestrating large scale ML workloads across a hybrid compute infrastructure of GPUs and CPUs. Your primary mission is to build a unified, elastic, and multi-tenant research substrate.

Key Responsibilities

• Elevating Researcher Experience: Design intuitive platform abstractions, APIs, scheduler wrappers, and a durable job control plane so researchers can seamlessly launch simulations, distributed training runs, and complex research pipelines without incurring infrastructure overhead
• Architecting Multi-Tenant Scheduling & Isolation: Build a multi-tenant compute substrate combining HPC-grade batch scheduling (gang scheduling, topology awareness, fair share) with cloud-native flexibility, enforcing strict tenant isolation across trading desks
• Enabling Multi-Datacenter & Multi-Cloud Compute Portability: Establish infrastructure agnostic execution abstractions that enable compute workloads to run transparently across multiple on-premise datacenters or burst into external cloud providers
• Standardizing Workflow Orchestration: Evaluate, select, and integrate production-grade job graph orchestration engines to automate multi-stage feature, training, and backtesting pipelines
• Building Fault-Tolerant Research Pipelines: Implement automated failure detection, retry-on-fault mechanisms, and high-performance checkpointing to ensure long-running distributed jobs survive hardware degradation and faults without manual intervention
• Optimizing Data-Paths: Profile and eliminate I/O bottlenecks, ensuring distributed ML workloads align with high-speed network fabrics and high-performance storage
• Delivering Observability & Cost Transparency: Implement comprehensive telemetry to track compute utilization, queue pressure, and GPU/CPU cost attribution, giving Portfolio Managers and Senior Management clear visibility into resource consumption and ROI

Technical Requirements

• HPC Administration: Deep experience with HPC job schedulers (e.g., Slurm), including gang scheduling, fair-share priority trees, topology-aware node allocation, and containerized HPC execution
• Kubernetes-native Engineering: Advanced understanding of Kubernetes architecture, CRDs, HPC focused operators, admission controllers and GPU-native schedulers
• Distributed ML Computing Frameworks: Familiarity with distributed computing frameworks (e.g., Ray) and deep learning frameworks (e.g., PyTorch, JAX) with multi-node scaling primitives
• Workflow Orchestration Expertise: Hands-on experience evaluating, architecting, and operating job graph orchestration frameworks
• Multi-Platform Architecture: Experience designing vendor-agnostic infrastructure layers, cloud-bursting strategies, and compute execution spanning multiple on-premise datacenters and public cloud environments
• Storage & Network Performance: Familiarity with high-performance storage solutions and high-speed network fabrics for large scale research workloads
• Compute Observability: Proven track record building cluster-wide telemetry and cost-attribution platforms for multi-tenant environments
• Architectural Leadership Mindset: A strong platform engineering mindset focused on reducing friction for researchers while maintaining rigorous operational efficiency, cost transparency, and system scalability

Comp: 300K+ base + Bonus