Senior Spark Acceleration Engineer (Kubernetes, Equity)

NVIDIA Corporation; United States; Remote; USD 184,000 - 288,000; Full time

Job summary

NVIDIA seeks a Senior Systems Software Engineer for the Apache Spark Acceleration group in the United States. You will design, build and operate a hosted Spark service control plane on Kubernetes, including the Java REST API, Spark Connect runner, Kyuubi, and Helm/Flux deployment.

You will own end-to-end services, extend cuDF Spark runtime and CPU/GPU workload router in Scala and Java, and collaborate with Delta Lake, Apache Iceberg and Unity Catalog teams.

Qualifications

• BS, MS, or PhD in Computer Science, Computer Engineering, or equivalent.
• 9+ years of experience in software development.
• Strong distributed-systems fundamentals; proficient in Java or Scala, plus C++ or Python.
• Hands-on experience operating services on Kubernetes, on-premise or cloud.
• Experience owning or reviewing long-lived servers: REST front ends, proxies or JDBC/Thrift services.
• Experience supporting enterprise production customers, including on-call.
• Experience using AI tools in software development.

Responsibilities

• Design, build and operate hosted Spark service control plane on Kubernetes: the Java REST API, the Spark Connect runner, Kyuubi, and deployment using Helm charts and Flux.
• Own code review for one or more services end to end and keep them releasable on the release train.
• Extend the cuDF Spark runtime and the CPU/GPU workload router in Scala and Java, including interoperability with Delta Lake, Apache Iceberg and Unity Catalog.
• Investigate production failures on clusters running more than a thousand concurrent jobs together with the customer team and upstream library teams, and drive fixes to closure.
• Instrument metrics, dashboards and alerts; take part in the on-call rotation and keep the runbooks current.
• Provide recommendations and feedback to teams regarding decisions surrounding topics such as infrastructure, continuous integration and testing strategy.

Skills

Distributed systems

Java/Scala

C++ or Python

Kubernetes

REST/Thrift

On-call support

AI tools in software dev

Education

BS/MS/PhD in Computer Science/Engineering or related field

Tools

Delta Lake

Apache Iceberg

Unity Catalog

Spark Connect

Kyuubi

Spark

Job description

NVIDIA seeks a Senior Systems Software Engineer for the Apache Spark Acceleration group in the United States. You will design, build and operate a hosted Spark service control plane on Kubernetes, including the Java REST API, Spark Connect runner, Kyuubi, and Helm/Flux deployment.

You will own end-to-end services, extend cuDF Spark runtime and CPU/GPU workload router in Scala and Java, and collaborate with Delta Lake, Apache Iceberg and Unity Catalog teams.