Senior Big Data Spark & Hadoop Administrator

Valiant Technologies LLC

Berkeley Heights (NJ)

On-site

USD 150,000 - 230,000

Full time

5 hours ago

Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Valiant Technologies LLC is seeking a senior big data engineer to deploy, configure, monitor, and troubleshoot Apache Spark clusters and related big data platforms. The role focuses on cluster health, performance tuning, and production data pipelines to ensure high availability and reliability.

You will work with Hadoop, Kafka, and cloud-based platforms, applying expertise in Spark architecture, cluster managers, and monitoring tools.

Qualifications

10+ years experience deploying, configuring, monitoring and troubleshooting Spark clusters.
Deep hands-on Spark architecture experience and familiarity with cluster managers (YARN/Kubernetes/Mesos).
Strong Linux/Unix administration and scripting skills (Bash, Python, Scala).
Experience with Hadoop, Kafka, Hive and monitoring suites (Splunk, Prometheus, Grafana).

Responsibilities

Cluster management: install, configure, and maintain Spark clusters and node infra in prod/staging.
Performance & monitoring: track cluster health, CPU/memory, and logs; optimize jobs.
Troubleshooting: diagnose OOMs, executor failures, shuffle issues, node crashes.
Pipeline & integration support: assist data engineers with Spark apps and streaming jobs.
Security & access control: implement RBAC, authentication, and data encryption.
Backup & disaster recovery: plan and execute backup, recovery, and failover for Spark apps.
Patching & upgrades: apply patches and driver updates with minimal disruption.