Data Engineer: Cloudera to AWS EMR Migration (Java/Spark)

Distillery; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Distillery is seeking a hands-on Data Engineer to migrate Java/Spark pipelines from Cloudera to AWS EMR and upgrade CDP workloads. You will also move orchestration from Airflow 2 to Airflow 3 while supporting streaming pipelines and building new ETL workflows.

You will work with a small team, ensuring pipelines are tested and running reliably in production, with at least 5 hours overlap with US Pacific Time. Immediate start on a long-term engagement is offered.

Qualifications

• 4+ years of experience building and running data pipelines in production.
• Strong Java skills with solid backend fundamentals.
• Hands-on experience with Apache Spark, preferably Spark batch jobs in Java.
• Working knowledge of the Hadoop ecosystem: HDFS, YARN, MapReduce, Hive or HBase.
• Experience scheduling and maintaining pipelines with Apache Airflow.
• Experience upgrading or migrating pipelines between platforms or versions.
• Professional English, both spoken and written.

Responsibilities

• Migrate Spark batch and MapReduce pipelines from Cloudera to AWS EMR, adapting code and logic.
• Upgrade HBase-dependent pipelines to the latest CDP version with Spark/Java upgrades.
• Validate migrated pipelines end to end for data parity and performance.
• Update Airflow DAGs and migrate orchestration from Airflow 2 to Airflow 3.
• Support a small number of streaming pipelines during migrations.
• Build new ETL pipelines based on stakeholder requests.
• Troubleshoot production issues and document migration steps.

Skills

Java

Backend engineering

Data pipelines

Tools

Apache Spark

Airflow

Hadoop ecosystem

Job description

Distillery is seeking a hands-on Data Engineer to migrate Java/Spark pipelines from Cloudera to AWS EMR and upgrade CDP workloads. You will also move orchestration from Airflow 2 to Airflow 3 while supporting streaming pipelines and building new ETL workflows.

You will work with a small team, ensuring pipelines are tested and running reliably in production, with at least 5 hours overlap with US Pacific Time. Immediate start on a long-term engagement is offered.