Hadoop Administrator

Arka Infotech Inc; San Francisco (CA); On-site; USD 120,000 - 150,000; Full time

Job summary

A tech company is looking for a Senior Software Engineer with 8 to 11 years of experience, focusing on Hadoop management and automation. You will be responsible for setting up and managing Hadoop clusters, providing 24x7 support, and performance tuning. Candidates should have an engineering degree or MCA, along with proficiency in data transfer techniques and troubleshooting Hadoop environments. This position is crucial for maintaining efficient production services in a dynamic tech setting.

Qualifications

• 8 – 11 years of experience in software engineering, particularly with Hadoop.
• Solid understanding of cluster management and monitoring practices.
• Experience with data ingestion techniques and debugging.

Responsibilities

• Set up production Hadoop clusters with optimum configurations.
• Drive automation of Hadoop deployments and maintenance.
• Provide 24x7 troubleshooting support for production services.
• Manage Hadoop clusters and monitor for alerts and issues.
• Performance tuning of Hadoop clusters.

Skills

Hadoop cluster management

Automation of deployments

Performance tuning

24x7 troubleshooting

Data transfer management

Education

Engineering Graduate or MCA

Job description

Job Title: Senior Software Engineer

Experience: 8 – 11 years

Location: SFO/ Houston

Primary Skills

• Set up production Hadoop clusters with optimum configurations
• Drive automation of Hadoop deployments, cluster expansion and maintenance operations
• Manage Hadoop cluster, monitoring alerts and notification
• Deployment of upgrades, updates and patches
• Provide 24x7 tier-3 troubleshooting and break-fix support for production services
• Diagnosis of installation & configuration issues
• Diagnosis of cluster management issues
• Diagnosis of performance issues
• Job scheduling, monitoring, debugging and troubleshooting
• Monitoring and management of the cluster in all respects, notably availability, performance and security
• File system management and monitoring
• Data transfer between Hadoop and other data stores (incl. relational database)
• Ingesting additional data sources into Hadoop in either streaming or batch mode
• Commissioning and decommissioning worker nodes
• Configuring the cluster to be rack‑aware
• Set up High Availability/Disaster Recovery environment
• Debug/Troubleshoot environment failures/downtime
• Working with data delivery teams to setup new Hadoop users. This job includes setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users
• Set up security configurations in Hadoop
• Performance tuning of Hadoop clusters and Hadoop Map Reduce routines
• Manage and review Hadoop log files

Qualifications

Engineering Graduate, MCA, Masters/Post Graduates (preferably IT/ CS)