Hadoop Admin

Skyleaf Consultants; Gurugram District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Skyleaf Consultants seeks a Platform Operations & Administration engineer to manage Hadoop ecosystem clusters, including HDFS, YARN, and MapReduce. You will handle provisioning, config, tuning, and day-to-day operational support to meet SLA targets.

You will also plan upgrades, coordinate releases, and manage runbooks, on-call readiness, and incident response while optimizing cluster performance and reliability.

Qualifications

• Experience administering Hadoop ecosystem clusters and related services.
• Knowledge of upgrade/patching lifecycle and rolling upgrades.

Responsibilities

• Administer and support Hadoop ecosystem clusters to meet SLA targets.
• Provision, configure, tune, and support day-to-day operations (start/stop, health checks, service validation).
• Manage multi-cluster environments with standardized configurations and procedures.
• Plan and execute platform upgrades and migrations with minimal downtime.
• Coordinate release rollouts with stakeholders and vendor tooling.
• Own runbooks, on-call readiness, and incident response processes.
• Tune HDFS/YARN and JVM for performance and stability.
• Forecast capacity and reliability planning for storage and compute.

Skills

Hadoop

HDFS

YARN

MapReduce

Cluster management

Performance tuning

Incident management

Root cause analysis

SLA targets

Tools

Cloudera

HDP

Apache

Job description

Job Description

Platform Operations & Administration

• Administer and support Hadoop ecosystem clusters (e.g., HDFS, YARN, MapReduce) and related services to meet SLA/SLI targets for availability and performance.
• Perform cluster provisioning, configuration, tuning, and day-to-day operational support (start/stop, health checks, service validation).
• Manage multi-cluster environments (dev/test/prod), including standardization of configurations and operational procedures.

Installation, Upgrades & Lifecycle Management

• Plan and execute platform upgrades, patching, and migrations with minimal downtime (including rolling upgrades where applicable).
• Maintain platform compatibility matrix, validate integrations, and coordinate release rollouts with stakeholders.
• Manage vendor distributions and tooling (e.g., Cloudera/HDP/Apache), including repository, parcels/packages, and version control.

Monitoring, Incident & Problem Management

• Implement robust monitoring/alerting for platform and infrastructure (CPU, memory, disk, network, I/O, JVM, services, queues, HDFS capacity, GC, etc.).
• Lead troubleshooting for incidents, perform root cause analysis (RCA), and drive corrective/preventive actions.
• Own operational runbooks, on-call readiness, escalation processes, and continuous improvement of response/MTTR.

Performance Tuning & Optimization

• Tune HDFS/YARN for workload performance and stability (scheduler/queue policies, resource limits, container sizing, node settings).
• Optimize cluster performance through JVM tuning, service parameters, OS/kernel tuning, and workload analysis.
• Identify and remediate bottlenecks (skew, small files, disk hotspots, network saturation, metadata pressure, etc.).

Capacity Planning & Reliability Engineering

• Forecast growth and perform capacity planning for storage, compute, and service scaling.
• Drive high availability and disaster recovery readiness (NameNode HA, ZK quorum health, backup/restore, replication, snapshots, DR drills).

Ensure data durability and operational resilience through proactive maintenance and reliability reviews.