Senior Databricks Engineer

Ethereum Technologies LLC

Austin (TX)

On-site

USD 85,000 - 116,000

Full time

33 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Ethereum Technologies in Austin, TX is seeking 5 Senior Databricks Engineers to migrate legacy Ab Initio ETL workflows into scalable PySpark pipelines on Databricks as part of a data modernization initiative. The team will design and implement modular pipelines, enable end-to-end data lineage, and ensure data quality across Snowflake and other sources.

The role requires strong Databricks/PySpark experience, ETL/ELT development expertise, and proficiency with enterprise scheduling (Control-M).

Qualifications

  • Strong hands-on Databricks and PySpark experience.
  • Experience migrating Ab Initio ETL workflows.
  • Experience with data lake architectures and Snowflake data integration.

Responsibilities

  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage and data quality governance artifacts.
  • Develop testing frameworks (unit/integration/reconciliation).
  • Perform dual-run comparisons between legacy and modernized pipelines.
  • Support UAT, pre-production validation, migration cutover, and decommissioning of legacy workflows.
  • Optimize job scheduling with tools like Control-M.
  • Prepare runbooks and monitoring/operational docs.

Skills

Databricks
PySpark
Ab Initio ETL
ETL/ELT
Snowflake
Data governance
Data quality
Unit testing
Control-M

Tools

Control-M
SQL
Snowflake

Job description

Job Title: Senior Databricks Engineer

Location: Austin, TX

Duration: 12 Months + Extension

Bill Rate: $73/hour

Job Type: W-2, 1099, C2C

Client: To Be Discussed Later

Work Authorization: US-Citizen, H-1B, OPT-EAD, GC-EAD

Job Description

We are hiring 5 Senior Databricks Engineers to support a large-scale data modernization initiative, migrating legacy Ab Initio ETL workflows into scalable, modular PySpark pipelines on Databricks.

Key Responsibilities:
  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage, data flow diagrams, and identify opportunities for process consolidation.
  • Implement data quality, governance, and validation standards.
  • Develop unit, integration, and reconciliation testing frameworks.
  • Perform dual-run comparisons between legacy and modernized pipelines.
  • Support UAT, pre-production validation, migration cutover, and legacy workflow decommissioning.
  • Optimize job scheduling using tools such as Control-M.
  • Prepare runbooks, monitoring solutions, and operational documentation.
Required Technical Skills:
  • Strong hands-on experience with Databricks and PySpark.
  • Experience in Ab Initio ETL migration or modernization.
  • Strong ETL/ELT development and data lake architecture experience.
  • Snowflake data integration experience.
  • Data lineage, data governance, and data quality.
  • Unit testing, integration testing, reconciliation, and dual-run validation.
  • Batch and near real-time data processing.
  • Control-M or similar enterprise scheduling tools.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks Engineer
Senior Databricks Engineer

Valiant Technologies LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Databricks Engineer
Senior Databricks Engineer

Quantum Technologies LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Databricks Engineer
Senior Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Databricks Engineer
Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

On-site
USD 84,000 - 120,000
Databricks Data Engineer
Databricks Data Engineer

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 170,000
Databricks Data Engineer
Databricks Data Engineer

Quantum Technologies LLC • Dallas (TX)

On-site
USD 150,000 - 190,000
Senior Databricks Developer
Senior Databricks Developer

Valiant Technologies LLC • Pennsylvania

On-site
USD 92,000 - 131,000
Senior Databricks Developer
Senior Databricks Developer

Ethereum Technologies LLC • Pennsylvania

On-site
USD 112,000 - 128,000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Valiant Technologies LLC • Sunnyvale (CA)

On-site
USD 83,000 - 103,000
Senior Data Engineer – Databricks & AWS
Senior Data Engineer – Databricks & AWS

Robotic Technologies Automation • Sunnyvale (CA)

On-site
USD 83,000 - 102,000