Data Engineer: Hadoop, Spark & Kafka (Onsite USA)

Pyramid Consulting, Inc; United States; On-site; MXN 1,242,000 - 1,490,000; Full time

Job summary

Pyramid Consulting, Inc. is urgently seeking a Hadoop Developer for a 12+ month contract located in Mexico/USA on-site. The role focuses on building scalable data processing systems using Hadoop, Spark, Kafka, and Airflow.

You will develop ETL pipelines, streaming solutions, and data ingestion workflows while collaborating with cross-functional teams. The candidate must relocate and work from the USA from day one, with compensation in the $50–$60/hour range and typical benefits.

Qualifications

• Experience with Hadoop ecosystem components (HDFS, YARN, Hive, HBase) and modern data platforms.
• Proficiency in Spark (SQL and Streaming), Kafka, Sqoop, and distributed ETL pipelines.
• Strong programming skills in Python/Scala/Java and shell scripting; experience with Airflow.
• Willingness to relocate and work onsite in the USA from Day 1.

Responsibilities

• Develop and maintain scalable data processing solutions using Hadoop ecosystem tools.
• Design and build distributed ETL pipelines for large-volume data processing.
• Develop Spark applications for batch and real-time processing.
• Build and manage Kafka-based data ingestion and streaming solutions.
• Utilize Sqoop to transfer data between relational databases and Hadoop stores.
• Create data processing scripts in Python/Scala/Java and shell scripts.
• Develop and orchestrate workflows with Airflow.
• Optimize jobs for performance, scalability, and reliability.
• Troubleshoot pipelines, Spark jobs, Hadoop components, and production issues.
• Collaborate with data engineers and technical teams for data integration and analytics.

Skills

Distributed data processing

ETL pipelines

Tools

Hadoop

Spark

Kafka

Sqoop

Python

Scala

Java

Shell scripting

Airflow

Job description

Pyramid Consulting, Inc. is urgently seeking a Hadoop Developer for a 12+ month contract located in Mexico/USA on-site. The role focuses on building scalable data processing systems using Hadoop, Spark, Kafka, and Airflow.

You will develop ETL pipelines, streaming solutions, and data ingestion workflows while collaborating with cross-functional teams. The candidate must relocate and work from the USA from day one, with compensation in the $50–$60/hour range and typical benefits.