Data Engineer (Java/Spark/Hadoop)

Distillery; United States; A distancia; USD 120.000 - 180.000; Jornada completa

Descripción de la vacante

Distillery is seeking a hands-on Data Engineer to migrate Java/Spark pipelines from Cloudera to AWS EMR and upgrade CDP workloads. You will also move orchestration from Airflow 2 to Airflow 3 while supporting streaming pipelines and building new ETL workflows.

You will work with a small team, ensuring pipelines are tested and running reliably in production, with at least 5 hours overlap with US Pacific Time. Immediate start on a long-term engagement is offered.

Formación

• 4+ years of experience building and running data pipelines in production.
• Strong Java skills with solid backend fundamentals.
• Hands-on experience with Apache Spark, preferably Spark batch jobs in Java.
• Working knowledge of the Hadoop ecosystem: HDFS, YARN, MapReduce, Hive or HBase.
• Experience scheduling and maintaining pipelines with Apache Airflow.
• Experience upgrading or migrating pipelines between platforms or versions.
• Professional English, both spoken and written.

Responsabilidades

• Migrate Spark batch and MapReduce pipelines from Cloudera to AWS EMR, adapting code and logic.
• Upgrade HBase-dependent pipelines to the latest CDP version with Spark/Java upgrades.
• Validate migrated pipelines end to end for data parity and performance.
• Update Airflow DAGs and migrate orchestration from Airflow 2 to Airflow 3.
• Support a small number of streaming pipelines during migrations.
• Build new ETL pipelines based on stakeholder requests.
• Troubleshoot production issues and document migration steps.

Conocimientos

Java

Backend engineering

Data pipelines

Herramientas

Apache Spark

Airflow

Hadoop ecosystem

Descripción del empleo

Distillery is a software development partner that helps organizations design, build, and scale modern technology solutions. Leading companies work with us to accelerate digital initiatives, solve complex technical challenges, and bring innovative products to market faster. Working as an extension of our clients' teams, we deliver lasting business value. Distillery is committed to diversity and inclusion. We embrace a dynamic and inclusive culture where you have the opportunity to continuously learn and grow.

About the Position

We are seeking a hands-on Data Engineer to join the data engineering team of a US-based technology client as the dedicated engineer for a set of platform migrations. In this role, you will move Java and Spark pipelines from Cloudera to AWS EMR, upgrade the remaining Cloudera workloads to the latest CDP version, and migrate orchestration from Airflow 2 to Airflow 3. Working with a small, experienced team that uses AI-assisted development tools day to day, you will execute the technical direction set by the client's staff engineers, making sure each pipeline is migrated, tested, and running reliably in production. This is a full-time, long-term position with immediate start, requiring at least 5 hours of overlap with US Pacific Time.

Responsibilities

• Migrate Spark batch and MapReduce pipelines written in Java from Cloudera to AWS EMR, adapting code, configuration, and business logic as needed.
• Upgrade HBase-dependent pipelines to the newest Cloudera CDP version, including Spark and Java version upgrades.
• Validate each migrated pipeline end to end, covering data parity, performance, and failure handling.
• Update Airflow DAGs and support the migration of orchestration from Airflow 2 to Airflow 3, ideally in the same pass as each pipeline migration.
• Support a small number of streaming pipelines through the same migrations.
• Build new ETL pipelines based on stakeholder requests.
• Troubleshoot and resolve production issues in existing pipelines.
• Document migration steps and findings so the rest of the team can reuse them.

Requirements

• 4+ years of experience in data engineering or backend software engineering, building and running data pipelines in production.
• Strong Java skills, with solid backend engineering fundamentals (build tools, dependency management, testing, and debugging).
• Hands-on experience with Apache Spark, ideally Spark batch jobs written in Java.
• Working knowledge of the Hadoop ecosystem: HDFS, YARN, MapReduce, Hive, or HBase.
• Experience scheduling and maintaining pipelines with Apache Airflow.
• Experience upgrading or migrating pipelines between platforms or framework versions, and validating that results remain unchanged.
• Ability to work independently, raise risks early, and keep work moving without close supervision.
• Professional English, both spoken and written.

Nice to Have

• Experience with AWS EMR, S3, and related AWS data services.
• Cloudera CDH or CDP administration or upgrade experience.
• Experience migrating from Airflow 2 to Airflow 3.
• Experience with Spark Structured Streaming or other streaming frameworks.
• Familiarity with DynamoDB, Redis, or other alternatives to HBase.
• Day-to-day use of AI coding tools such as Cursor or Claude.