Hadoop Administrator

Paytm; Dadri; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A technology company in Uttar Pradesh is looking for an experienced Hadoop Administrator to manage and maintain on-premises Hadoop clusters. The role includes cluster administration, performance optimization, and ensuring high availability of services. The ideal candidate should have hands-on experience with Apache Hadoop, YARN, and Hive, as well as expertise in Linux administration. This position also involves troubleshooting cluster issues and implementing backup and disaster recovery strategies.

Qualifications

• Strong hands-on experience with Apache Hadoop versions 2.7.x and 3.x.
• Expertise in managing Hadoop clusters including HDFS and YARN.
• Experience with Linux administration (RHEL/CentOS/Ubuntu) and shell scripting.

Responsibilities

• Install, configure, and manage on-premises Hadoop clusters.
• Administer and monitor HDFS including storage management and data balancing.
• Manage resource allocation and scheduling using Apache YARN.
• Optimize Apache Hive, ensuring high availability of critical services.
• Troubleshoot and support Hadoop cluster issues.

Skills

Apache Hadoop version 2.7.x and 3.x

Apache YARN

Apache Hive

HDFS administration

Linux administration

Shell scripting

Job description

Job Description: Hadoop Administrator

We are looking for an experienced Hadoop Administrator to manage, maintain, and upgrade our on-premises Hadoop clusters. The role involves working on Apache Hadoop versions 2.7.1 and 3.4.x, ensuring cluster stability, performance, and scalability, along with ecosystem components like Hive and YARN.

Key Responsibilities

1. Cluster Administration

• Install, configure, and manage on-prem Hadoop clusters based on Apache Hadoop (versions 2.7.1 and 3.4.x).
• Administer and monitor Hadoop Distributed File System (HDFS) including storage management, replication, and data balancing.
• Manage resource allocation and scheduling using Apache YARN.

2. Upgrade & Migration

• Hadoop upgrades (2.7 → 3.4) in on‑prem environments.
• Perform cluster migration, data validation (HDFS checksums, block integrity), and backward compatibility validation for jobs.
• Experience with migration tools and distribution changes.

3. Ecosystem Management

• Administer and optimize Apache Hive (query tuning, metastore management).
• Work with ecosystem tools like Spark (basic admin exposure preferred) & Tez (execution engine for Hive).
• Ensure high availability for Hive Metastore and other critical services.

4. Performance & Optimization

• Monitor cluster health, job performance, and resource utilization.
• Tune YARN queues and capacity scheduler, Hive queries and execution plans, disk, memory, and network utilization.
• Implement performance improvements for large-scale data processing.

5. Security & Governance

• Configure authentication/authorization.
• Ensure data security and compliance policies.
• Manage access controls and audit logs.

6. Troubleshooting & Support

• Provide support for Hadoop cluster issues.
• Troubleshoot node failures, job failures (Hive/YARN) & data inconsistencies.
• Perform log analysis and root cause identification.

7. Backup & DR

• Plan and manage backup and disaster recovery strategies.
• Implement DistCp-based replication & DR cluster setup (active-passive / active-active).

Required Skills

• Strong hands‑on experience with Apache Hadoop (2.7.x and 3.x).
• Expertise in Apache YARN, Apache Hive & HDFS administration.
• Experience in on-prem cluster setup and hardware-level optimization.
• Linux administration (RHEL/CentOS/Ubuntu).
• Shell scripting for automation.