Senior Databricks Engineer - PySpark & Data Modernization

Robotic Process Automation LLC

Austin (TX)

Presencial

USD 83 000 - 118 000

Tempo integral

Há 5 dias
Torna-te num dos primeiros candidatos

Recebe uma resposta deste recrutador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

Robotic Technologies Automation in Austin, TX seeks 5 Senior Databricks Engineers to modernize data pipelines by migrating legacy Ab Initio ETL workflows to scalable PySpark pipelines on Databricks. You will design and optimize end-to-end ETL/ELT processes, integrate Snowflake, and ensure data quality and governance.

Responsibilities include implementing unit and integration tests, building data lineage diagrams, supporting UAT and cutovers, and refining job scheduling with Control-M.

Qualificações

  • 5+ years of data engineering experience focused on ETL/ELT.
  • Proficient in Databricks and PySpark.
  • Experience migrating Ab Initio ETL workflows.
  • Strong Snowflake data integration and data governance.

Responsabilidades

  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage, data flow diagrams, and identify opportunities for process consolidation.
  • Implement data quality, governance, and validation standards.
  • Develop unit, integration, and reconciliation testing frameworks.
  • Perform dual-run comparisons between legacy and modernized pipelines.
  • Support UAT, pre-production validation, migration cutover, and legacy workflow decommissioning.
  • Optimize job scheduling using tools such as Control-M.
  • Prepare runbooks, monitoring solutions, and operational documentation.

Conhecimentos

Databricks
PySpark
Ab Initio
ETL/ELT
Snowflake
Data governance
Unit/integration testing
Control-M

Ferramentas

Control-M

Descrição da oferta de emprego

Robotic Technologies Automation in Austin, TX seeks 5 Senior Databricks Engineers to modernize data pipelines by migrating legacy Ab Initio ETL workflows to scalable PySpark pipelines on Databricks. You will design and optimize end-to-end ETL/ELT processes, integrate Snowflake, and ensure data quality and governance.

Responsibilities include implementing unit and integration tests, building data lineage diagrams, supporting UAT and cutovers, and refining job scheduling with Control-M.

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Databricks Engineer - Data Modernization & PySpark
Senior Databricks Engineer - Data Modernization & PySpark

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer - Modern Data Pipelines PySpark
Senior Databricks Engineer - Modern Data Pipelines PySpark

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000
Senior Databricks Engineer: Data Modernization & PySpark
Senior Databricks Engineer: Data Modernization & PySpark

Quantum Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Databricks & PySpark Engineer — Data Modernization
Databricks & PySpark Engineer — Data Modernization

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Data Engineer - Spark, AWS & Delta Lake
Senior Databricks Data Engineer - Spark, AWS & Delta Lake

Robotic Technologies Automation • Dallas (TX)

Presencial
USD 140 000 - 180 000
Senior Databricks Engineer
Senior Databricks Engineer

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Quantum Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000
Senior Databricks Engineer
Senior Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000