Senior Databricks Engineer

CloudBigD

Austin (TX)

Presencial

USD 83 000 - 118 000

Tempo integral

há 5 horas
Torna-te num dos primeiros candidatos

Uma candidatura completa num minuto

Currículo e carta de apresentação personalizados, prontos a enviar.

Ultrapassa os filtros ATS

Resumo da oferta

CloudBigD in Austin, TX is seeking 5 Senior Databricks Engineers to modernize data pipelines by migrating legacy Ab Initio ETL to PySpark on Databricks. You will design scalable ETL/ELT processes, refactor monolithic workflows, and integrate Snowflake with robust data quality and governance.

The role covers batch and near real-time processing, data lineage, testing frameworks, and operation runbooks. A strong background in data engineering and enterprise scheduling is required for a successful

Qualificações

  • Strong hands-on Databricks and PySpark experience.
  • Experience migrating Ab Initio ETL workloads.
  • ETL/ELT development for data lakes.
  • Snowflake integration experience.
  • Data lineage, governance, and data quality.
  • Unit testing, integration testing, reconciliation, dual-run validation.
  • Batch and near real-time processing.
  • Familiarity with enterprise scheduling tools like Control-M.

Responsabilidades

  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage and governance artifacts.
  • Implement data quality and validation standards.
  • Develop unit/integration testing frameworks and run reconciliations.
  • Prepare runbooks, monitoring, and operational documentation.

Conhecimentos

Databricks
PySpark
ETL/ELT development
Data lake architecture
Data governance
Data quality
Unit testing
Integration testing
Data lineage

Ferramentas

Snowflake
Control-M

Descrição da oferta de emprego

Job Title: Senior Databricks Engineer

Location: Austin, TX

Duration: 12 Months + Extension

Bill Rate: $73/hour

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US-Citizen, H-1B, OPT-EAD, GC-EAD

Job Description

We are hiring 5 Senior Databricks Engineers to support a large-scale data modernization initiative, migrating legacy Ab Initio ETL workflows into scalable, modular PySpark pipelines on Databricks.

Key Responsibilities:
  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage, data flow diagrams, and identify opportunities for process consolidation.
  • Implement data quality, governance, and validation standards.
  • Develop unit, integration, and reconciliation testing frameworks.
  • Perform dual-run comparisons between legacy and modernized pipelines.
  • Support UAT, pre-production validation, migration cutover, and legacy workflow decommissioning.
  • Optimize job scheduling using tools such as Control-M.
  • Prepare runbooks, monitoring solutions, and operational documentation.
Required Technical Skills:
  • Strong hands-on experience with Databricks and PySpark.
  • Experience in Ab Initio ETL migration or modernization.
  • Strong ETL/ELT development and data lake architecture experience.
  • Snowflake data integration experience.
  • Data lineage, data governance, and data quality.
  • Unit testing, integration testing, reconciliation, and dual-run validation.
  • Batch and near real-time data processing.
  • Control-M or similar enterprise scheduling tools.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Databricks Engineer
Senior Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Quantum Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Robotics Technologies • Sunnyvale (CA)

Presencial
USD 76 000 - 108 000
Databricks Engineer
Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 84 000 - 120 000
Databricks Data Engineer
Databricks Data Engineer

Ethereum Technologies LLC • Dallas (TX)

Presencial
USD 120 000 - 170 000
Databricks Data Engineer
Databricks Data Engineer

Quantum Technologies LLC • Dallas (TX)

Presencial
USD 150 000 - 190 000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Robotic Process Automation LLC • Sunnyvale (CA)

Presencial
USD 83 000 - 102 000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Quantum Technologies LLC • Sunnyvale (CA)

Presencial
USD 76 000 - 108 000