Hadoop Administrator

Impetus Technologies; Bengaluru; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Impetus Technologies in Bengaluru seeks a hands-on Big Data Engineer to manage HDFS storage, NiFi data ingestion, and Kubernetes-based platforms. The role involves ensuring cluster health on the HPE Ezmeral stack, addressing security events, and supporting Hive and data workloads.

Expect involvement in CI/CD pipelines with GitHub and ArgoCD. Ideal candidates have practical experience with NiFi, Kafka, Iceberg, Trino/Starburst, ElasticSearch and MinIO, plus Kubernetes administration.

Qualifications

• Hands-on exposure to listed skills, willing to work at client office 3 days/week.

Responsibilities

• Managing HDFS storage, disk utilization and data availability to maintain cluster health mainly HPE Ezemeral platform.

Skills

BigData / Hadoop Administration

Linux

SQL

Data Engineering

Apache NiFi

Kafka

Iceberg

Trino

Starburst

HPE Ezmeral

HDFS

Yarn RM

ElasticSearch

MinIO

Kubernetes

ServiceNow

Jira

GitHub

ArgoCD

Copilot

Tools

HPE Ezmeral

Kafka

NiFi

Trino

Starburst

Elastic/Kibana

Grafana

GitHub

ArgoCD

Job description

Candidate should have hands-on exposure on given skills and is willing to work at client office 3 days/week

• BigData / Hadoop Administration
• Programming & Scripting: Linux, SQL
• Data Engineering & Processing: Apache NiFi, Kafka, Iceberg, Trino, Starburst
• Big Data Ecosystem: HPE Ezmeral, HDFS, Yarn RM
• Search & Analytics: ElasticSearch
• Storage & Infrastructure: MinIO, Kubernetes
• CI/CD & Collaboration Tools: ServiceNow, Jira, GitHub, ArgoCD, Copilot

Roles & Responsibilities

Roles & Responsibilities:

• Managing HDFS storage, disk utilization and data availability to maintain cluster health mainly HPE Ezemeral platform.
• Managing security-related activities, including service accounts password resets and certificate renewals
• Supporting Hive and Big Data workloads, troubleshooting job failures for downstream NiFi ingestion like to ssh failures/krb issues.
• Supporting and performing HPE Ezemeral server reboots and ensure high availability of the system during such maintenance activities
• Monitoring YARN ResourceManager/NodeManager and cluster resource utilization, troubleshooting application and resource-related issues.
• Manage end-to-end production incident and change management, including monitoring, troubleshooting, RCA, and SLA adherence.
• Manage and maintain Apache NiFi data ingestion workflows across multiple environments, ensuring reliable pipeline execution and resolving operational issues.
• Support and troubleshoot Kubernetes-based data platforms, ensuring cluster stability, availability, and performance.
• Manage Kafka streaming pipelines, including monitoring, troubleshooting data issues, consumer lag, and pipeline backlogs.
• Manage Trino/Starburst and Iceberg-based data lakehouse platforms
• Manage MinIO object storage supporting data lake and ingestion pipelines.
• Support SIEM data management
• Manage CI/CD deployments and configuration management using GitHub and ArgoCD across environments.
• Perform platform maintenance and security operations, certificate/svc ID renewals, authentication-related activities, and scheduled maintenance.
• Monitor platform and pipeline health using Grafana and Elastic/Kibana, proactively identifying and resolving operational issues.
• Support platform/data migrations, including environment assessment, dependency mapping, documentation.