ETL Modernization Developer - 1645

aKube Inc; United States; Remote; USD 79,000 - 107,000; Full time

Job summary

aKube Inc is seeking a remote Data Engineer to assess and migrate IBM DataStage workloads to Databricks Spark on AWS. You will design and implement cloud-based ETL pipelines, validate data quality, and build automated testing and deployment processes.

The role requires PySpark, Python, SQL, Shell scripting, Delta Lake, Unity Catalog, and Photon, with CI/CD using GitLab or Azure DevOps. Nice-to-have: Glue, Lambda, Redshift, and data governance experience.

Qualifications

• Strong hands-on experience developing and maintaining IBM DataStage jobs.
• Proven experience modernizing large-scale ETL/ELT and data warehouse workloads.
• Experience building Databricks and Spark pipelines in an AWS environment.
• Advanced programming skills in PySpark, Python, SQL, and Shell scripting.
• Experience with Delta Lake, Unity Catalog, and Databricks Photon.
• Experience with GitLab or Azure DevOps for CI/CD.
• Knowledge of automated testing frameworks and ETL validation methods.
• Experience supporting production cutovers, stabilization, and application teams.

Responsibilities

• Assess existing ETL/ELT jobs, databases, and data warehouses for migration readiness.
• Re-engineer legacy DataStage jobs using Databricks Spark on AWS.
• Develop, test, and deploy scalable cloud-based data pipelines.
• Perform parity checks, unit testing, functional testing, UAT, regression testing, and performance testing.
• Validate schemas, data quality, pipeline performance, and migration accuracy.
• Build CI/CD processes for automated development, testing, and deployment.
• Support migration cutover, production deployment, and hypercare.
• Coordinate application redeployment and end-to-end testing.
• Decommission legacy DataStage jobs and related artifacts.
• Prepare technical documentation and operational readiness materials.
• Conduct knowledge-transfer and training sessions for engineering and business teams.

Skills

IBM DataStage development

ETL/ELT migration

Databricks Spark

PySpark

Python

SQL

Shell scripting

Delta Lake

Unity Catalog

Photon

CI/CD

ETL testing

Tools

Databricks

Spark on AWS

AWS Glue

Lambda

Redshift

GitLab

Azure DevOps

PyTest

Job description

Onsite/Hybrid/Remote: Remote Duration: 12 months Rate Range: $68 in W2 Work Authorization: GC and US Citizens Only Must Have: IBM DataStage development ETL/ELT migration and modernization Databricks and Apache Spark on AWS PySpark, Python, SQL, and Shell scripting Delta Lake, Unity Catalog, and Photon AWS Glue, Lambda, and Redshift CI/CD and automated ETL testing PyTest and XML/JSON parsing

Responsibilities

• Assess existing ETL/ELT jobs, databases, and data warehouses for migration readiness.
• Re-engineer legacy IBM DataStage jobs using Databricks Spark on AWS.
• Develop, test, and deploy scalable cloud-based data pipelines.
• Perform parity checks, unit testing, functional testing, UAT, regression testing, and performance testing.
• Validate schemas, data quality, pipeline performance, and migration accuracy.
• Build CI/CD processes for automated development, testing, and deployment.
• Support migration cutover, production deployment, and hypercare.
• Coordinate application redeployment and end-to-end testing.
• Decommission legacy DataStage jobs and related artifacts.
• Prepare technical documentation and operational readiness materials.
• Conduct knowledge-transfer and training sessions for engineering and business teams.

Qualifications

• Strong hands-on experience developing and maintaining IBM DataStage jobs.
• Proven experience modernizing large-scale ETL/ELT and data warehouse workloads.
• Experience building Databricks and Spark pipelines in an AWS environment.
• Advanced programming skills in PySpark, Python, SQL, and Shell scripting.
• Experience with Delta Lake, Unity Catalog, and Databricks Photon.
• Experience with GitLab or Azure DevOps for CI/CD.
• Knowledge of automated testing frameworks and ETL validation methods.
• Experience supporting production cutovers, stabilization, and application teams.

Nice to Have

• Experience with AWS Glue, Lambda, and Redshift.
• Experience with JIRA and Agile delivery.
• Knowledge of data governance and cloud security standards.
• Experience creating operational documentation and delivering technical training.

Originally posted on Himalayas