Senior Spark & Hadoop Cluster Admin

Machine Learning Technologies, LLC; Berkeley Heights (NJ); On-site; USD 150,000 - 210,000; Full time

Job summary

Machine Learning Technologies, LLC seeks an experienced Spark infrastructure engineer to deploy, configure, monitor, and troubleshoot Apache Spark clusters and related big data platforms in production environments.

You will own cluster health, performance tuning, and data pipelines, while collaborating with data engineers and developers on Spark applications, streaming jobs with Kafka, and integration with monitoring suites like Splunk, Prometheus, and Grafana.

Qualifications

• Experience deploying and maintaining Spark clusters in production.
• Strong Linux/Unix scripting and automation skills.
• Familiarity with monitoring and log-analysis tools (Splunk, Prometheus, Grafana).

Responsibilities

• Maintain cluster health, optimize performance, and ensure data pipeline reliability.
• Work with Spark architecture components: Driver, Executors, RDDs/DataFrames.
• Monitor resource usage and troubleshoot OOM, failures, and bottlenecks.
• Support data engineers and developers with Spark apps and Kafka integrations.
• Implement security, access control, and disaster recovery for clusters.
• Collaborate with stakeholders and communicate findings clearly.

Skills

Spark architecture

Cluster health monitoring

Linux scripting

Big data tooling

Debugging logs

Stakeholder communication

Tools

Spark UI

Splunk

Prometheus

Grafana

Kafka

Hadoop

Hive

Kubernetes

YARN

Job description

Machine Learning Technologies, LLC seeks an experienced Spark infrastructure engineer to deploy, configure, monitor, and troubleshoot Apache Spark clusters and related big data platforms in production environments.

You will own cluster health, performance tuning, and data pipelines, while collaborating with data engineers and developers on Spark applications, streaming jobs with Kafka, and integration with monitoring suites like Splunk, Prometheus, and Grafana.