Hadoop Developer

Pyramid Consulting, Inc; United States; On-site; MXN 1,242,000 - 1,490,000; Full time

Job summary

Pyramid Consulting, Inc. is urgently seeking a Hadoop Developer for a 12+ month contract located in Mexico/USA on-site. The role focuses on building scalable data processing systems using Hadoop, Spark, Kafka, and Airflow.

You will develop ETL pipelines, streaming solutions, and data ingestion workflows while collaborating with cross-functional teams. The candidate must relocate and work from the USA from day one, with compensation in the $50–$60/hour range and typical benefits.

Qualifications

• Experience with Hadoop ecosystem components (HDFS, YARN, Hive, HBase) and modern data platforms.
• Proficiency in Spark (SQL and Streaming), Kafka, Sqoop, and distributed ETL pipelines.
• Strong programming skills in Python/Scala/Java and shell scripting; experience with Airflow.
• Willingness to relocate and work onsite in the USA from Day 1.

Responsibilities

• Develop and maintain scalable data processing solutions using Hadoop ecosystem tools.
• Design and build distributed ETL pipelines for large-volume data processing.
• Develop Spark applications for batch and real-time processing.
• Build and manage Kafka-based data ingestion and streaming solutions.
• Utilize Sqoop to transfer data between relational databases and Hadoop stores.
• Create data processing scripts in Python/Scala/Java and shell scripts.
• Develop and orchestrate workflows with Airflow.
• Optimize jobs for performance, scalability, and reliability.
• Troubleshoot pipelines, Spark jobs, Hadoop components, and production issues.
• Collaborate with data engineers and technical teams for data integration and analytics.

Skills

Distributed data processing

ETL pipelines

Tools

Hadoop

Spark

Kafka

Sqoop

Python

Scala

Java

Shell scripting

Airflow

Job description

Immediate need for a talented Hadoop Developer This is a 12+ months opportunity with long-term potential and is located in Mexico/United States (onsite)

Job Diva ID: 26-24992

Pay Range: $50 - $60/hour. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).

Key Responsibilities

• Note- Candidates must be willing to relocate and work onsite in the USA from Day 1
• Develop and maintain scalable data processing solutions using the Hadoop ecosystem, including HDFS, YARN, Hive, and HBase.
• Design and develop distributed ETL pipelines for large-volume data processing and transformation.
• Develop Spark applications using Spark SQL and Spark Streaming for batch and real-time data processing.
• Build and maintain Kafka-based data ingestion and streaming solutions.
• Use Sqoop to transfer data between relational databases and Hadoop environments.
• Develop applications and data processing scripts using Python, Scala, and/or Java.
• Create Shell scripts to automate data processing and operational tasks.
• Develop and manage Airflow workflows for data pipeline orchestration and scheduling.
• Optimize distributed processing jobs for performance, scalability, and reliability.
• Troubleshoot data pipelines, Spark jobs, Hadoop components, and production data-processing issues.
• Collaborate with data engineers and technical teams to support data integration and large-scale analytics requirements.

Key Requirements and Technology Experience

• Must have skills: - Hadoop Ecosystem (HDFS, YARN, Hive, HBase), Spark (SQL, Streaming), Kafka, Sqoop, Python/Scala/Java, Shell scripting, Airflow (workflow orchestration), distributed data processing, ETL pipelines

Our client is a leading IT Industry, and we are currently interviewing to fill this and other similar contract positions.

Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, colour, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.

By applying to our jobs, you agree to receive calls, AI-generated calls, text messages, or emails from Pyramid Consulting, Inc. and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy here.