Remote ETL Modernization Engineer (Databricks on AWS)

aKube Inc; United States; Remote; USD 79,000 - 107,000; Full time

Job summary

aKube Inc is seeking a remote Data Engineer to assess and migrate IBM DataStage workloads to Databricks Spark on AWS. You will design and implement cloud-based ETL pipelines, validate data quality, and build automated testing and deployment processes.

The role requires PySpark, Python, SQL, Shell scripting, Delta Lake, Unity Catalog, and Photon, with CI/CD using GitLab or Azure DevOps. Nice-to-have: Glue, Lambda, Redshift, and data governance experience.

Qualifications

• Strong hands-on experience developing and maintaining IBM DataStage jobs.
• Proven experience modernizing large-scale ETL/ELT and data warehouse workloads.
• Experience building Databricks and Spark pipelines in an AWS environment.
• Advanced programming skills in PySpark, Python, SQL, and Shell scripting.
• Experience with Delta Lake, Unity Catalog, and Databricks Photon.
• Experience with GitLab or Azure DevOps for CI/CD.
• Knowledge of automated testing frameworks and ETL validation methods.
• Experience supporting production cutovers, stabilization, and application teams.

Responsibilities

• Assess existing ETL/ELT jobs, databases, and data warehouses for migration readiness.
• Re-engineer legacy DataStage jobs using Databricks Spark on AWS.
• Develop, test, and deploy scalable cloud-based data pipelines.
• Perform parity checks, unit testing, functional testing, UAT, regression testing, and performance testing.
• Validate schemas, data quality, pipeline performance, and migration accuracy.
• Build CI/CD processes for automated development, testing, and deployment.
• Support migration cutover, production deployment, and hypercare.
• Coordinate application redeployment and end-to-end testing.
• Decommission legacy DataStage jobs and related artifacts.
• Prepare technical documentation and operational readiness materials.
• Conduct knowledge-transfer and training sessions for engineering and business teams.

Skills

IBM DataStage development

ETL/ELT migration

Databricks Spark

PySpark

Python

SQL

Shell scripting

Delta Lake

Unity Catalog

Photon

CI/CD

ETL testing

Tools

Databricks

Spark on AWS

AWS Glue

Lambda

Redshift

GitLab

Azure DevOps

PyTest

Job description

aKube Inc is seeking a remote Data Engineer to assess and migrate IBM DataStage workloads to Databricks Spark on AWS. You will design and implement cloud-based ETL pipelines, validate data quality, and build automated testing and deployment processes.

The role requires PySpark, Python, SQL, Shell scripting, Delta Lake, Unity Catalog, and Photon, with CI/CD using GitLab or Azure DevOps. Nice-to-have: Glue, Lambda, Redshift, and data governance experience.