Bigdata /Hadoop/Spark Admin

Envision Technology Solutions; Berkeley Heights (NJ); On-site; USD 150,000 - 190,000; Full time

Job summary

Envision Technology Solutions seeks an experienced Apache Spark / Big Data Administrator to lead the deployment, configuration, and monitoring of large Spark clusters in our Berkeley Heights, NJ onsite environment. You will focus on cluster health, performance tuning, and reliable production data pipelines.

The role requires deep hands-on Spark architecture knowledge, familiarity with YARN/Kubernetes/ Mesos, and strong Linux scripting skills.

Qualifications

• 10+ years of experience deploying and managing Spark clusters.

Responsibilities

• Cluster management: Install, configure, and maintain Apache Spark clusters and node infrastructure in production and staging environments.
• Performance & monitoring: Monitor cluster health, resource utilization (CPU, memory), and execution logs; optimize job execution times.
• Troubleshooting: Diagnose and resolve out-of-memory errors, executor failures, and shuffle issues in distributed workloads.
• Pipeline & integration support: Support data engineers and developers in managing Spark applications, streaming jobs, and integrations with Kafka.
• Security & access control: Implement security principles, RBAC, authentication, and data encryption across the cluster.
• Backup & disaster recovery: Execute backup, recovery, and failover plans for Spark applications and configurations.

Skills

Apache Spark expertise

Big data frameworks

Cluster health & perf tuning

Linux/Unix administration

Bash/Python/Scala scripting

Production data pipelines

Tools

Hadoop

Kafka

Hive

Splunk

Prometheus

Grafana

Job description

Location: Berkley Heights, NJ Onsite

Job Description

• 10+ years solid experience with deploying, configuring, monitoring, and troubleshooting Apache Spark clusters and associated big data frameworks (such as Hadoop, Kafka, or cloud-based data platforms) to ensure high availability and performance.
• Focuses on cluster health, performance tuning, and production data pipelines.
• Technical Experience: Deep hands-on experience with Apache Spark architecture (Driver, Workers, Executors, RDDs/DataFrames) and cluster managers (YARN, Kubernetes, or Mesos).
• OS & Scripting: Strong proficiency in Linux/Unix administration and scripting languages (Bash, Python, or Scala).
• Ecosystem Knowledge: Familiarity with big data and streaming tools like Hadoop HDFS, Kafka, Hive, or monitoring suites (Splunk, Prometheus, Grafana).
• Debugging Skills: Strong analytical capabilities to inspect logs, thread dumps, and metrics under heavy production loads
• Excellent coordination with stakeholders and communication skills

Role: Apache Spark / Big Data Administrator

Location: Berkley Heights, NJ Onsite

Job Description

• 10+ years solid experience with deploying, configuring, monitoring, and troubleshooting Apache Spark clusters and associated big data frameworks (such as Hadoop, Kafka, or cloud-based data platforms) to ensure high availability and performance.
• Focuses on cluster health, performance tuning, and production data pipelines.
• Technical Experience: Deep hands-on experience with Apache Spark architecture (Driver, Workers, Executors, RDDs/DataFrames) and cluster managers (YARN, Kubernetes, or Mesos).
• OS & Scripting: Strong proficiency in Linux/Unix administration and scripting languages (Bash, Python, or Scala).
• Ecosystem Knowledge: Familiarity with big data and streaming tools like Hadoop HDFS, Kafka, Hive, or monitoring suites (Splunk, Prometheus, Grafana).
• Debugging Skills: Strong analytical capabilities to inspect logs, thread dumps, and metrics under heavy production loads
• Excellent coordination with stakeholders and communication skills

Must Have Experience With Below Responsibilities

• Cluster Management: Install, configure, and maintain Apache Spark clusters and node infrastructure in production and staging environments.
• Performance & Monitoring: Monitor cluster health, resource utilization (CPU, memory), and execution logs using tools like Spark UI, Splunk, or Elastic. Identify bottlenecks and optimize job execution times.
• Troubleshooting: Diagnose and resolve out-of-memory errors, executor failures, shuffle issues, and node crashes in distributed workloads.
• Pipeline & Integration Support: Support data engineers and developers in managing Spark applications, streaming jobs, and integrations with message brokers like Kafka.
• Security & Access Control: Implement security principles, role-based access control, authentication, and data encryption across the cluster.
• Backup & Disaster Recovery: Execute backup, recovery, and failover plans for Spark applications, configurations, and critical data stores.

Patching & Upgrades

Apply software patches, driver updates, and version upgrades with minimal disruption to ongoing production jobs