Senior Spark & Big Data Platform Engineer

KTek Resourcing LLC; Berkeley Heights (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

KTek Resourcing LLC is seeking a senior Big Data Engineer to deploy, configure and maintain Apache Spark clusters in production and staging environments in Berkeley Heights, NJ. You will focus on cluster health, performance optimization, and building robust data pipelines with Hadoop, Kafka, and related platforms.

Responsibilities include monitoring with Spark UI, Splunk, and Elastic; diagnosing out-of-memory errors and node crashes; implementing security, DR plans, and routine patching with

Qualifications

• 10+ years deploying and supporting Spark clusters in production.
• Experience with Hadoop, Kafka, and cloud data platforms.
• Strong Linux/Unix, Bash, Python, or Scala scripting.

Responsibilities

• Cluster Management: Install, configure, and maintain Apache Spark clusters and node infrastructure in production and staging environments.
• Performance & Monitoring: Monitor cluster health, resources, and logs using Spark UI, Splunk, or Elastic; identify bottlenecks and optimize job times.
• Troubleshooting: Diagnose and resolve out-of-memory errors, executor failures, shuffle issues, and node crashes in distributed workloads.
• Pipeline & Integration Support: Support data engineers and developers in managing Spark apps, streaming jobs, and Kafka integrations.
• Security & Access Control: Implement security principles, RBAC, authentication, and data encryption across the cluster.
• Backup & Disaster Recovery: Execute backup, recovery, and failover plans for Spark apps and configs.
• Patching & Upgrades: Apply software patches and driver updates with minimal disruption.

Skills

Spark cluster

Cluster management

Performance tuning

Linux administration

Scripting: Bash

Python

Scala

Kafka

Hadoop

Monitoring: Splunk

Prometheus

Grafana

Communication

Troubleshooting

Data pipelines

Tools

Spark UI

YARN

Kubernetes

Mesos

HDFS

Kafka

Elastic

Splunk

Prometheus

Grafana

Job description

KTek Resourcing LLC is seeking a senior Big Data Engineer to deploy, configure and maintain Apache Spark clusters in production and staging environments in Berkeley Heights, NJ. You will focus on cluster health, performance optimization, and building robust data pipelines with Hadoop, Kafka, and related platforms.

Responsibilities include monitoring with Spark UI, Splunk, and Elastic; diagnosing out-of-memory errors and node crashes; implementing security, DR plans, and routine patching with