Senior Databricks Engineer - Modern Data Pipelines PySpark

Ethereum Technologies LLC

Austin (TX)

Presencial

USD 85 000 - 116 000

Tempo integral

Há 5 dias
Torna-te num dos primeiros candidatos

Recebe uma resposta deste recrutador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

Ethereum Technologies in Austin, TX is seeking 5 Senior Databricks Engineers to migrate legacy Ab Initio ETL workflows into scalable PySpark pipelines on Databricks as part of a data modernization initiative. The team will design and implement modular pipelines, enable end-to-end data lineage, and ensure data quality across Snowflake and other sources.

The role requires strong Databricks/PySpark experience, ETL/ELT development expertise, and proficiency with enterprise scheduling (Control-M).

Qualificações

  • Strong hands-on Databricks and PySpark experience.
  • Experience migrating Ab Initio ETL workflows.
  • Experience with data lake architectures and Snowflake data integration.

Responsabilidades

  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage and data quality governance artifacts.
  • Develop testing frameworks (unit/integration/reconciliation).
  • Perform dual-run comparisons between legacy and modernized pipelines.
  • Support UAT, pre-production validation, migration cutover, and decommissioning of legacy workflows.
  • Optimize job scheduling with tools like Control-M.
  • Prepare runbooks and monitoring/operational docs.

Conhecimentos

Databricks
PySpark
Ab Initio ETL
ETL/ELT
Snowflake
Data governance
Data quality
Unit testing
Control-M

Ferramentas

Control-M
SQL
Snowflake

Descrição da oferta de emprego

Ethereum Technologies in Austin, TX is seeking 5 Senior Databricks Engineers to migrate legacy Ab Initio ETL workflows into scalable PySpark pipelines on Databricks as part of a data modernization initiative. The team will design and implement modular pipelines, enable end-to-end data lineage, and ensure data quality across Snowflake and other sources.

The role requires strong Databricks/PySpark experience, ETL/ELT development expertise, and proficiency with enterprise scheduling (Control-M).

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Databricks Engineer - Data Modernization & PySpark
Senior Databricks Engineer - Data Modernization & PySpark

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer - PySpark & Data Modernization
Senior Databricks Engineer - PySpark & Data Modernization

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer: Data Modernization & PySpark
Senior Databricks Engineer: Data Modernization & PySpark

Quantum Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Databricks & PySpark Engineer — Data Modernization
Databricks & PySpark Engineer — Data Modernization

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000
Senior Databricks Engineer
Senior Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Quantum Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Data Engineer - Spark, PySpark & AWS
Senior Databricks Data Engineer - Spark, PySpark & AWS

Ethereum Technologies • Dallas (TX)

Presencial
USD 120 000 - 170 000