Onsite Hadoop & Spark Data Engineer: ETL, Kafka, Airflow

Pyramid Consulting, Inc; North Carolina; On-site; USD 69,000 - 83,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Paid sick leave

Job summary

Pyramid Consulting, Inc. is seeking a Hadiio Developer for a 12+ month contract located onsite in the United States. You will design and implement large-scale data processing frameworks using the Hadoop ecosystem, Spark, Kafka, and Airflow to support real-time and batch analytics.

Ideal candidates have strong Python/Scala/Java skills and hands-on experience with ETL pipelines, Sqoop, and distributed systems. This role requires onsite work from Day 1 and offers comprehensive benefits.

Qualifications

• Must have deep hands-on experience with Hadoop ecosystem components (HDFS, YARN, Hive, HBase) and Spark.
• Strong background building and optimizing ETL pipelines and data processing jobs.
• Experience with streaming data and distributed data platforms is required.

Responsibilities

• Note: Candidates must be willing to work onsite from Day 1.
• Develop and maintain scalable data processing solutions using Hadoop ecosystem, including HDFS, YARN, Hive, and HBase.
• Design and develop distributed ETL pipelines for large-volume data processing and transformation.
• Develop Spark applications using Spark SQL and Spark Streaming for batch and real-time data processing.
• Build and maintain Kafka-based data ingestion and streaming solutions.
• Use Sqoop to transfer data between relational databases and Hadoop environments.
• Develop applications and data processing scripts using Python, Scala, and/or Java.
• Create Shell scripts to automate data processing and operational tasks.
• Develop and manage Airflow workflows for data pipeline orchestration and scheduling.
• Optimize distributed processing jobs for performance, scalability, and reliability.
• Troubleshoot data pipelines, Spark jobs, Hadoop components, and production data-processing issues.
• Collaborate with data engineers and technical teams to support data integration and large-scale analytics requirements.

Skills

Hadoop ecosystem

Spark (SQL/Streaming)

Kafka

Sqoop

Python/Scala/Java

Shell scripting

Airflow

ETL pipelines

Job description

Pyramid Consulting, Inc. is seeking a Hadiio Developer for a 12+ month contract located onsite in the United States. You will design and implement large-scale data processing frameworks using the Hadoop ecosystem, Spark, Kafka, and Airflow to support real-time and batch analytics.

Ideal candidates have strong Python/Scala/Java skills and hands-on experience with ETL pipelines, Sqoop, and distributed systems. This role requires onsite work from Day 1 and offers comprehensive benefits.