Senior Spark & Hadoop Cluster Engineer

Ethereum Technologies; Berkeley Heights (NJ); On-site; USD 180,000 - 280,000; Full time

Job summary

Ethereum Technologies is seeking a senior Big Data Engineer with 10+ years of hands-on experience deploying, configuring, monitoring, and troubleshooting Apache Spark clusters to ensure high availability and peak performance across production and staging environments.

You will own cluster health and performance tuning, manage security and access controls, oversee upgrades and DR plans, and collaborate with data engineers on Spark applications, Kafka integrations, and cloud data platforms to

Qualifications

• 10+ years solid experience with deploying, configuring, monitoring, and troubleshooting Apache Spark clusters and related big data platforms.

Responsibilities

• Cluster Management: Install, configure, and maintain Spark clusters and node infrastructure in production and staging environments.
• Performance & Monitoring: Monitor cluster health, resource utilization, and execution logs using Spark UI, Splunk, or Elastic.
• Troubleshooting: Diagnose and resolve out-of-memory errors, executor failures, shuffle issues, and node crashes in distributed workloads.
• Pipeline & Integration Support: Support data engineers and developers in managing Spark applications, streaming jobs, and Kafka integrations.
• Security & Access Control: Implement RBAC, authentication, and data encryption across the cluster.
• Backup & Disaster Recovery: Execute backup, recovery, and failover plans for Spark applications and configurations.
• Patching & Upgrades: Apply software patches and driver updates with minimal disruption to production jobs.

Skills

Spark architecture

Cluster health monitoring

Linux administration

Scripting languages

Analytical debugging

Communication skills

Tools

Spark UI

Splunk

Prometheus

Grafana

YARN

Kubernetes

Mesos

Kafka

Hadoop

Job description

Ethereum Technologies is seeking a senior Big Data Engineer with 10+ years of hands-on experience deploying, configuring, monitoring, and troubleshooting Apache Spark clusters to ensure high availability and peak performance across production and staging environments.

You will own cluster health and performance tuning, manage security and access controls, oversee upgrades and DR plans, and collaborate with data engineers on Spark applications, Kafka integrations, and cloud data platforms to