Senior Spark & Hadoop Cluster Engineer

Quantum Technologies. LLC; New Jersey; On-site; USD 120,000 - 180,000; Full time

Job summary

Quantum Technologies. LLC seeks a highly skilled data engineering specialist to deploy, configure, monitor, and troubleshoot Apache Spark clusters and associated big data frameworks.

The role emphasizes high availability, performance, and secure access across production environments. You will focus on cluster health, performance tuning, and supporting data engineers with streaming pipelines and integrations with Kafka, Hadoop, and cloud platforms.

Qualifications

• Expertise deploying and managing Spark clusters in production.
• Experience with Hadoop, Kafka, and cloud data platforms.
• Strong Linux/Unix scripting (Bash, Python, Scala).

Responsibilities

• Maintain Spark clusters for high availability and performance.
• Tune performance, monitor pipelines, and investigate bottlenecks.
• Troubleshoot OOMs, executor failures, and shuffle issues.
• Collaborate with data engineering and DevOps teams on deployments.
• Ensure security and access control across the cluster.

Skills

Spark architecture

Linux scripting

Python

Scala

Kafka

Hadoop

Monitoring tools

Troubleshooting

Stakeholder communication

Tools

Spark UI

Kubernetes

YARN

HDFS

Prometheus

Grafana

Splunk

Elastic

Job description

Quantum Technologies. LLC seeks a highly skilled data engineering specialist to deploy, configure, monitor, and troubleshoot Apache Spark clusters and associated big data frameworks.

The role emphasizes high availability, performance, and secure access across production environments. You will focus on cluster health, performance tuning, and supporting data engineers with streaming pipelines and integrations with Kafka, Hadoop, and cloud platforms.