Bigdata Hadoop Administrator

MACHINE LEARNING TECHNOLOGIES LLC; Florida; On-site; USD 140,000 - 210,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC seeks an experienced Bigdata Hadoop Administrator in Berkeley Heights, NJ to deploy and manage large Spark clusters, ensure availability and performance in production environments. This 16-month role requires deep technical hands-on expertise across Hadoop, Spark, Kafka, and cloud-based data platforms.

You will optimize pipelines, monitor resources, and collaborate with data engineers and security teams to implement access controls and disaster recovery plans.

Qualifications

• 10+ years solid experience deploying, configuring, monitoring Spark clusters.
• Linux/Unix administration and scripting (Bash, Python, Scala) proficiency.
• Experience with Hadoop ecosystem (Hadoop HDFS, Kafka, Hive) and monitoring suites.

Responsibilities

• Focus on cluster health, performance tuning, and production pipelines.
• Install, configure, and maintain Spark clusters and node infra.
• Monitor health and resource utilization using Spark UI, Splunk, Elastic.
• Debug logs, optimize jobs, and collaborate with data engineers.
• Ensure security and data governance across clusters.
• Coordinate with stakeholders and communicate status.

Skills

Apache Spark

Linux/Unix administration

Scripting (Bash/Python/Scala)

Kafka

Hadoop

Kubernetes

Monitoring tools (Splunk/Prometheus/GF

Data pipelines

Tools

Hadoop HDFS

Kafka

Spark UI

Splunk

Prometheus

Grafana

Hive

Job description

Job Description

• Job ID: J53565
• Job Title: Bigdata Hadoop Administrator
• Location: Berkley Heights, NJ
• Duration: 16 Months + Extension
• Domain Exposure: Manufacturing
• Work Authorization: US Citizen,Green Card,OPT-EAD,CPT,H-1B,H4-EAD,L2-EAD,GC-EAD
• Client: To Be Discussed Later
• Employment Type: W-2, 1099, C2C

Job Summary

10+ years solid experience with deploying, configuring, monitoring, and troubleshooting Apache Spark clusters and associated big data frameworks (such as Hadoop, Kafka, or cloud-based data platforms) to ensure high availability and performance.

Key Responsibilities

• Focuses on cluster health, performance tuning, and production data pipelines.
• Technical Experience: Deep hands-on experience with Apache Spark architecture (Driver, Workers, Executors, RDDs/Data Frames) and cluster managers (YARN, Kubernetes, or Mesos).
• OS & Scripting: Strong proficiency in Linux/Unix administration and scripting languages (Bash, Python, or Scala).
• Ecosystem Knowledge: Familiarity with big data and streaming tools like Hadoop HDFS, Kafka, Hive, or monitoring suites (Splunk, Prometheus, Grafana).
• Debugging Skills: Strong analytical capabilities to inspect logs, thread dumps, and metrics under heavy production loads
• Excellent coordination with stakeholders and communication skills

Must Have Experience With Below Responsibilities

• Cluster Management: Install, configure, and maintain Apache Spark clusters and node infrastructure in production and staging environments.
• Performance & Monitoring: Monitor cluster health, resource utilization (CPU, memory), and execution logs using tools like Spark UI, Splunk, or Elastic. Identify bottlenecks and optimize job execution times.
• Troubleshooting: Diagnose and resolve out-of-memory errors, executor failures, shuffle issues, and node crashes in distributed workloads.
• Pipeline & Integration Support: Support data engineers and developers in managing Spark applications, streaming jobs, and integrations with message brokers like Kafka.
• Security & Access Control: Implement security principles, role-based access control, authentication, and data encryption across the cluster.
• Backup & Disaster Recovery: Execute backup, recovery, and failover plans for Spark applications, configurations, and critical data stores.
• Patching & Upgrades: Apply software patches, driver updates, and version upgrades with minimal disruption to ongoing production jobs

Equal Opportunity Employer

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.