Senior Spark & Big Data Admin - Cluster Health

Envision Technology Solutions

Berkeley Heights (NJ)

On-site

USD 150,000 - 190,000

Full time

8 hours ago

Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Envision Technology Solutions seeks an experienced Apache Spark / Big Data Administrator to lead the deployment, configuration, and monitoring of large Spark clusters in our Berkeley Heights, NJ onsite environment. You will focus on cluster health, performance tuning, and reliable production data pipelines.

The role requires deep hands-on Spark architecture knowledge, familiarity with YARN/Kubernetes/ Mesos, and strong Linux scripting skills.

Qualifications

10+ years of experience deploying and managing Spark clusters.

Responsibilities

Cluster management: Install, configure, and maintain Apache Spark clusters and node infrastructure in production and staging environments.
Performance & monitoring: Monitor cluster health, resource utilization (CPU, memory), and execution logs; optimize job execution times.
Troubleshooting: Diagnose and resolve out-of-memory errors, executor failures, and shuffle issues in distributed workloads.
Pipeline & integration support: Support data engineers and developers in managing Spark applications, streaming jobs, and integrations with Kafka.
Security & access control: Implement security principles, RBAC, authentication, and data encryption across the cluster.
Backup & disaster recovery: Execute backup, recovery, and failover plans for Spark applications and configurations.