Senior GPU Systems Engineer - Spark Acceleration on Kubernetes

NVIDIA Corporation; Champaign (IL); On-site; USD 184,000 - 288,000; Full time

Job summary

NVIDIA is seeking a Senior Systems Software Engineer for the Apache Spark Acceleration group, based in Champaign, IL. You will design, build and operate hosted Spark service control plane on Kubernetes, extending the cuDF Spark runtime, and coordinating with Delta Lake, Apache Iceberg and Unity Catalog.

You will own end-to-end services, review code, investigate production issues, and contribute to scalability and reliability of thousands of jobs per hour on Kubernetes and AWS, with on-call

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or a closely related field (or equivalent experience).
• 9+ years of experience in software development.
• Strong distributed-systems fundamentals and proficiency in Java or Scala, plus C++ or Python.
• Hands-on experience operating services on Kubernetes, on-premise or cloud.
• Experience owning or reviewing a long-lived server: REST front ends, proxies or request routers, JDBC or Thrift services.
• Experience supporting enterprise or internal production customers, including on-call.
• Experience using AI tools in software development.

Responsibilities

• Design, build and operate hosted Spark service control plane on Kubernetes: the Java REST API, the Spark Connect runner, Kyuubi, and deployment using Helm charts and Flux
• Own code review for one or more services end to end and keep them releasable on the release train
• Extend the cuDF Spark runtime and the CPU/GPU workload router in Scala and Java, including interoperability with Delta Lake, Apache Iceberg and Unity Catalog
• Investigate production failures on clusters running more than a thousand concurrent jobs together with the customer team and upstream library teams, and drive fixes to closure
• Instrument metrics, dashboards and alerts; take part in the on-call rotation and keep the runbooks current
• Provide recommendations and feedback to teams regarding decisions surrounding topics such as infrastructure, continuous integration and testing strategy

Skills

Distributed systems

Java

Scala

C++

Python

Education

BS/MS/PhD in Computer Science or related field

Tools

Kubernetes

Helm

Flux

Spark Connect

Kyuubi

Job description

NVIDIA is seeking a Senior Systems Software Engineer for the Apache Spark Acceleration group, based in Champaign, IL. You will design, build and operate hosted Spark service control plane on Kubernetes, extending the cuDF Spark runtime, and coordinating with Delta Lake, Apache Iceberg and Unity Catalog.

You will own end-to-end services, review code, investigate production issues, and contribute to scalability and reliability of thousands of jobs per hour on Kubernetes and AWS, with on-call