Senior Hadoop Developer

Veriipro; Charlotte (NC); On-site; USD 120,000 - 150,000; Full time

Job summary

Veriipro is seeking a senior data platform engineer to own and optimize on-premise Hadoop ecosystems in the Charlotte area. The role focuses on maintaining HDFS, Hive Metastore, Spark and Ozone, and ensuring data lifecycle governance.

You will lead troubleshooting, capacity planning, data ingestion, ETL processes, and automation using Shell and Python, collaborating with cross-functional teams to keep production systems reliable and scalable.

Qualifications

• 10+ years of experience in Hadoop ecosystem and large-scale distributed data platforms.
• Strong hands-on experience with HDFS, Hive, Spark and Ozone.
• Experience with Hive Metastore administration and table/partition maintenance.
• Experience with Hadoop data lifecycle management, storage management, and data retention.
• SQL, Shell scripting, and Python automation proficiency.
• Experience with CDP or similar Hadoop distributions.
• Strong data analysis, troubleshooting, and root cause analysis skills.

Responsibilities

• Monitor, troubleshoot, and resolve HDFS and Apache Ozone issues.
• Investigate Hadoop cluster performance, storage utilization, capacity, and availability.
• Manage storage allocation, quota requests, capacity planning, and storage forecasting.
• Perform Hive Metastore administration, including table and partition maintenance.
• Support Hadoop platform maintenance, upgrades, integrations, onboarding, and decommissioning.
• Support data ingestion, ETL workflows, and data validation processes.
• Develop and maintain Shell and Python automation scripts for operational activities.
• Collaborate with application, data engineering, and platform teams during incident, problem, and change management activities.
• Perform root cause analysis and implement corrective and preventive actions.
• Create and maintain runbooks, operational documentation, knowledge articles, and support procedures.
• Ensure effective Hadoop data lifecycle, storage, and quota governance.

Tools

HDFS

Apache Hive

Apache Spark

Apache Ozone

Hive Metastore

Data lifecycle management

Storage management

Data retention

SQL

Shell scripting

Python automation

Cloudera Data Platform

CDP or Hadoop distributions

Monitoring/Observability tools

Job description

Experience Required: 10+ Years

Required Technical/Functional Skills

• 10+ years of experience working with the Hadoop ecosystem and large-scale distributed data platforms.
• Strong hands-on experience with:
  • HDFS
  • Apache Hive
  • Apache Spark
  • Apache Ozone
• Strong experience with Hive Metastore management, including table and partition maintenance.
• Experience with Hadoop data lifecycle management, storage management, and data retention.
• Strong knowledge of data ingestion, ETL processes, data validation, and troubleshooting.
• Proficiency in SQL, Shell scripting, and Python automation.
• Experience with Cloudera Data Platform (CDP) or similar Hadoop distributions.
• Strong data analysis, troubleshooting, and root cause analysis skills.

Roles & Responsibilities

• Monitor, troubleshoot, and resolve HDFS and Apache Ozone issues.
• Investigate Hadoop cluster performance, storage utilization, capacity, and availability concerns.
• Manage storage allocation, quota requests, capacity planning, and storage forecasting.
• Perform Hive Metastore administration, including table and partition maintenance.
• Support Hadoop platform maintenance, upgrades, integrations, onboarding, and decommissioning activities.
• Support data ingestion, ETL workflows, and data validation processes.
• Develop and maintain Shell and Python automation scripts for operational activities.
• Collaborate with application, data engineering, and platform teams during incident, problem, and change management activities.
• Perform root cause analysis and implement corrective and preventive actions.
• Create and maintain runbooks, operational documentation, knowledge articles, and support procedures.
• Ensure effective Hadoop data lifecycle, storage, and quota governance.

Preferred / Desired Skills

• Experience supporting large-scale, multi-petabyte Hadoop environments.
• Strong understanding of distributed storage architecture, replication, and fault tolerance.
• Experience with capacity management, storage forecasting, quota governance, and utilization optimization.
• Experience with Hadoop platform onboarding, integration, migration, and decommissioning.
• Familiarity with AWS, Azure, or GCP cloud platforms.
• Experience with monitoring and observability tools such as Cloudera Manager, Grafana, Splunk, and Prometheus.
• Strong analytical, troubleshooting, and problem-solving skills.
• Experience performing Root Cause Analysis (RCA) for complex production issues.
• Strong communication and collaboration skills.