Hadoop Developer

Pyramid Consulting, Inc; North Carolina; On-site; USD 69,000 - 83,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Paid sick leave

Job summary

Pyramid Consulting, Inc. is seeking a Hadiio Developer for a 12+ month contract located onsite in the United States. You will design and implement large-scale data processing frameworks using the Hadoop ecosystem, Spark, Kafka, and Airflow to support real-time and batch analytics.

Ideal candidates have strong Python/Scala/Java skills and hands-on experience with ETL pipelines, Sqoop, and distributed systems. This role requires onsite work from Day 1 and offers comprehensive benefits.

Qualifications

• Must have deep hands-on experience with Hadoop ecosystem components (HDFS, YARN, Hive, HBase) and Spark.
• Strong background building and optimizing ETL pipelines and data processing jobs.
• Experience with streaming data and distributed data platforms is required.

Responsibilities

• Note: Candidates must be willing to work onsite from Day 1.
• Develop and maintain scalable data processing solutions using Hadoop ecosystem, including HDFS, YARN, Hive, and HBase.
• Design and develop distributed ETL pipelines for large-volume data processing and transformation.
• Develop Spark applications using Spark SQL and Spark Streaming for batch and real-time data processing.
• Build and maintain Kafka-based data ingestion and streaming solutions.
• Use Sqoop to transfer data between relational databases and Hadoop environments.
• Develop applications and data processing scripts using Python, Scala, and/or Java.
• Create Shell scripts to automate data processing and operational tasks.
• Develop and manage Airflow workflows for data pipeline orchestration and scheduling.
• Optimize distributed processing jobs for performance, scalability, and reliability.
• Troubleshoot data pipelines, Spark jobs, Hadoop components, and production data-processing issues.
• Collaborate with data engineers and technical teams to support data integration and large-scale analytics requirements.

Skills

Hadoop ecosystem

Spark (SQL/Streaming)

Kafka

Sqoop

Python/Scala/Java

Shell scripting

Airflow

ETL pipelines

Job description

Immediate need for a talented Hadiio Developer. This is a 12+ months contract opportunity with long-term potential and is located in United States (onsite).

Job Diva Id- 26-24992

Pay Range: $50 - $60/hour. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).

Key Responsibilities

• Note: Candidates must be willing to work onsite from Day 1
• Develop and maintain scalable data processing solutions using the Hadoop ecosystem, including HDFS, YARN, Hive, and HBase.
• Design and develop distributed ETL pipelines for large-volume data processing and transformation.
• Develop Spark applications using Spark SQL and Spark Streaming for batch and real-time data processing.
• Build and maintain Kafka-based data ingestion and streaming solutions.
• Use Sqoop to transfer data between relational databases and Hadoop environments.
• Develop applications and data processing scripts using Python, Scala, and/or Java.
• Create Shell scripts to automate data processing and operational tasks.
• Develop and manage Airflow workflows for data pipeline orchestration and scheduling.
• Optimize distributed processing jobs for performance, scalability, and reliability.
• Troubleshoot data pipelines, Spark jobs, Hadoop components, and production data-processing issues.
• Collaborate with data engineers and technical teams to support data integration and large-scale analytics requirements.

Key Requirements and Technology Experience

• Must have skills: - Hadoop Ecosystem (HDFS, YARN, Hive, HBase), Spark (SQL, Streaming), Kafka, Sqoop, Python/Scala/Java, Shell scripting, Airflow (workflow orchestration), distributed data processing, ETL pipelines

Our client is a leading IT Industry, and we are currently interviewing to fill this and other similar contract positions.

Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, colour, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.