Senior Databricks Engineer: Data Modernization & PySpark

Quantum Technologies LLC

Austin (TX)

Presencial

USD 83 000 - 118 000

Tempo integral

Há 6 dias
Torna-te num dos primeiros candidatos

Recebe uma resposta deste recrutador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

Quantum Technologies LLC in Austin, TX is seeking 5 Senior Databricks Engineers to support a large-scale data modernization initiative, 12 Months + Extension. You will migrate legacy Ab Initio ETL workflows into scalable PySpark pipelines on Databricks, shaping the future of our data platform.

Responsibilities include designing and optimizing data pipelines, building end-to-end data lineage, and ensuring data quality and governance.

Qualificações

  • Strong hands-on experience with Databricks and PySpark.
  • Experience in Ab Initio ETL migration or modernization.
  • Strong ETL/ELT development and data lake architecture experience.
  • Snowflake data integration experience.
  • Data lineage, data governance, and data quality.
  • Unit testing, integration testing, reconciliation, and dual-run validation.
  • Batch and near real-time data processing.
  • Control-M or similar enterprise scheduling tools.

Responsabilidades

  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage, data flow diagrams, and identify opportunities for process consolidation.
  • Implement data quality, governance, and validation standards.
  • Develop unit, integration, and reconciliation testing frameworks.
  • Perform dual-run comparisons between legacy and modernized pipelines.
  • Support UAT, pre-production validation, migration cutover, and legacy workflow decommissioning.
  • Optimize job scheduling using tools such as Control-M.
  • Prepare runbooks, monitoring solutions, and operational documentation.

Conhecimentos

Databricks
PySpark
Ab Initio
ETL/ELT development
Data lake architecture
Snowflake integration
Data governance
Data quality
Testing frameworks
Batch processing

Ferramentas

Control-M

Descrição da oferta de emprego

Quantum Technologies LLC in Austin, TX is seeking 5 Senior Databricks Engineers to support a large-scale data modernization initiative, 12 Months + Extension. You will migrate legacy Ab Initio ETL workflows into scalable PySpark pipelines on Databricks, shaping the future of our data platform.

Responsibilities include designing and optimizing data pipelines, building end-to-end data lineage, and ensuring data quality and governance.

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Databricks Engineer - Data Modernization & PySpark
Senior Databricks Engineer - Data Modernization & PySpark

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer - Modern Data Pipelines PySpark
Senior Databricks Engineer - Modern Data Pipelines PySpark

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000
Databricks & PySpark Engineer — Data Modernization
Databricks & PySpark Engineer — Data Modernization

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer - PySpark & Data Modernization
Senior Databricks Engineer - PySpark & Data Modernization

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Data Engineer: Scale Big Data Pipelines
Senior Databricks Data Engineer: Scale Big Data Pipelines

Quantum Technologies. LLC • Dallas (TX)

Presencial
USD 150 000 - 190 000
Senior Databricks Engineer
Senior Databricks Engineer

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks & AWS Data Engineer
Senior Databricks & AWS Data Engineer

Quantum Technologies LLC • Sunnyvale (CA)

Presencial
USD 76 000 - 108 000
Senior Databricks Engineer
Senior Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000