Senior Spark & Hadoop Admin for High-Performance Pipelines

MACHINE LEARNING TECHNOLOGIES LLC; Florida; Presencial; USD 140.000 - 210.000; Jornada completa

Descripción de la vacante

MACHINE LEARNING TECHNOLOGIES LLC seeks an experienced Bigdata Hadoop Administrator in Berkeley Heights, NJ to deploy and manage large Spark clusters, ensure availability and performance in production environments. This 16-month role requires deep technical hands-on expertise across Hadoop, Spark, Kafka, and cloud-based data platforms.

You will optimize pipelines, monitor resources, and collaborate with data engineers and security teams to implement access controls and disaster recovery plans.

Formación

• 10+ years solid experience deploying, configuring, monitoring Spark clusters.
• Linux/Unix administration and scripting (Bash, Python, Scala) proficiency.
• Experience with Hadoop ecosystem (Hadoop HDFS, Kafka, Hive) and monitoring suites.

Responsabilidades

• Focus on cluster health, performance tuning, and production pipelines.
• Install, configure, and maintain Spark clusters and node infra.
• Monitor health and resource utilization using Spark UI, Splunk, Elastic.
• Debug logs, optimize jobs, and collaborate with data engineers.
• Ensure security and data governance across clusters.
• Coordinate with stakeholders and communicate status.

Conocimientos

Apache Spark

Linux/Unix administration

Scripting (Bash/Python/Scala)

Kafka

Hadoop

Kubernetes

Monitoring tools (Splunk/Prometheus/GF

Data pipelines

Herramientas

Hadoop HDFS

Kafka

Spark UI

Splunk

Prometheus

Grafana

Hive

Descripción del empleo

MACHINE LEARNING TECHNOLOGIES LLC seeks an experienced Bigdata Hadoop Administrator in Berkeley Heights, NJ to deploy and manage large Spark clusters, ensure availability and performance in production environments. This 16-month role requires deep technical hands-on expertise across Hadoop, Spark, Kafka, and cloud-based data platforms.

You will optimize pipelines, monitor resources, and collaborate with data engineers and security teams to implement access controls and disaster recovery plans.