Senior Hadoop & Data Infrastructure Engineer

Polus Solutions Pvt. Ltd.; Hinoba-an; On-site; PHP 1,800,000 - 3,000,000; Full time

Job summary

Polus Solutions Pvt. Ltd. is seeking a Senior Hadoop Administrator & Data Engineer to manage and optimize Hadoop clusters, including HDP/CDH environments and Spark workloads.

You’ll tune HDFS/YARN, supervise HiveServer2/Metastore, and administer Ambari, with strong Linux skills and scripting to support resilient data infrastructure. You will partner with data engineers to keep Airflow/AWS MWAA pipelines healthy, enforce capacity planning, and handle upgrades across on-prem and cloud-integrated

Qualifications

• 4+ years administering production Hadoop clusters (HDP, CDH, or equivalent).
• Deep, hands-on HDFS, YARN, and Hive administration — not just usage.
• Strong Linux systems administration: SSH, systemd, and troubleshooting on RHEL/CentOS-family hosts.
• Spark-on-YARN operational experience with memory tuning and driver issues.
• Scripting proficiency in Bash and Python.
• Experience with Ambari administration and in-place Hadoop version upgrades.
• Airflow/AWS MWAA experience for data pipelines.
• MySQL and MongoDB operations at scale.
• Elasticsearch or Sphinx/Manticore search infrastructure familiarity.
• Terraform and AWS (S3, EMR, IAM) expertise.
• Experience decommissioning or migrating legacy Hadoop estates.

Responsibilities

• Operate and tune HDFS, YARN, Hive, and Spark across clusters with mixed-version estates.
• Manage gateway host capacity and prevent resource starvation.
• Run Hive Metastore and HiveServer2 including JVM sizing and error diagnosis.
• Administer clusters via Ambari: configs, alerts, and upgrades.
• Plan capacity, quotas, and data retention with documented cleanup policies.
• Collaborate with data engineering teams and support remote pipelines via SSH.
• Maintain access control, SSH hygiene, and service accounts across nodes.
• Build monitoring and alerting for host load, per-tenant usage, and metastore health.

Skills

HDFS administration

YARN administration

Hive administration

Linux administration

Bash scripting

Python scripting

Ambari administration

Apache Airflow

AWS MWAA

Terraform

SQL Databases (MySQL)

NoSQL (MongoDB)

Elasticsearch

Sphinx/Manticore

AWS Core services (S3, EMR, IAM)

Hadoop version migration

Tools

Hadoop (HDP/CDH)

Spark

Ambari

Airflow/AWS MWAA

MySQL

MongoDB

Elasticsearch

Terraform

Job description

Polus Solutions Pvt. Ltd. is seeking a Senior Hadoop Administrator & Data Engineer to manage and optimize Hadoop clusters, including HDP/CDH environments and Spark workloads.

You’ll tune HDFS/YARN, supervise HiveServer2/Metastore, and administer Ambari, with strong Linux skills and scripting to support resilient data infrastructure. You will partner with data engineers to keep Airflow/AWS MWAA pipelines healthy, enforce capacity planning, and handle upgrades across on-prem and cloud-integrated