RL AI Infrastructure Engineer

Innova software Services Inc; Fremont (CA); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Innova software Services Inc. is seeking an AI Lab Tech Engineer to own the execution layer beneath reinforcement learning environments. You will build scalable, secure sandboxing and high-performance execution for production workloads, collaborating with research, data teams, and enterprise clients.

The role emphasizes long-horizon, multi-node scaling, cost-aware optimization, and robust observability. A hybrid work model (2 days onsite, 3 days remote) is supported in Fremont, CA.

Qualifications

• Production distributed systems experience.
• Containerization and isolation tech (namespaces, cgroups, VMs, gVisor, Firecracker).
• Proficiency in Python; experience with Rust, Go, or C++.
• Cloud & on-prem deployments on AWS, GCP and enterprise hardware.
• Performance profiling, scheduling, and cost optimization.
• Strong communication with technical stakeholders and enterprise clients.

Responsibilities

• Environment Execution & Sandboxing: design/maintain isolation and execution layer for RL environments; snapshot/restore state to pause, resume, inspect, branch rollouts.
• Fault Detection & Recovery: automate failure detection, revert to known-good states, patch and resume execution.
• Long-Horizon & Multi-Node Scaling: extend execution to multi-node environments across tools/services for hours/days.
• Performance & Cost Optimization: drive throughput/latency, manage scheduling and usage to maximize rollouts per dollar.
• Observability & Profiling: identify bottlenecks from container init to teardown; build deep observability for thousands of runs.
• Environment Platform Engineering: build frameworks/tools to package/deploy RL environments; provide debugging tooling.
• Model Deployment: deploy/manage AI models on on-premises hardware.
• Production Standards: scale prototypes into production-grade systems with tests, validation, and docs.

Skills

Systems & Infrastruktur

Low-Level Systems

Languages & Tools

Cloud & On-Premises

Performance Engineering

Soft Skills

Tools

Claude Code

Job description

Innova software Services Inc. is seeking an AI Lab Tech Engineer to own the execution layer beneath reinforcement learning environments. You will build scalable, secure sandboxing and high-performance execution for production workloads, collaborating with research, data teams, and enterprise clients.

The role emphasizes long-horizon, multi-node scaling, cost-aware optimization, and robust observability. A hybrid work model (2 days onsite, 3 days remote) is supported in Fremont, CA.