Senior Databricks Engineer - Data Modernization & PySpark

CloudBigD

Austin (TX)

On-site

USD 83,000 - 118,000

Full time

4 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

CloudBigD in Austin, TX is seeking 5 Senior Databricks Engineers to modernize data pipelines by migrating legacy Ab Initio ETL to PySpark on Databricks. You will design scalable ETL/ELT processes, refactor monolithic workflows, and integrate Snowflake with robust data quality and governance.

The role covers batch and near real-time processing, data lineage, testing frameworks, and operation runbooks. A strong background in data engineering and enterprise scheduling is required for a successful

Qualifications

  • Strong hands-on Databricks and PySpark experience.
  • Experience migrating Ab Initio ETL workloads.
  • ETL/ELT development for data lakes.
  • Snowflake integration experience.
  • Data lineage, governance, and data quality.
  • Unit testing, integration testing, reconciliation, dual-run validation.
  • Batch and near real-time processing.
  • Familiarity with enterprise scheduling tools like Control-M.

Responsibilities

  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage and governance artifacts.
  • Implement data quality and validation standards.
  • Develop unit/integration testing frameworks and run reconciliations.
  • Prepare runbooks, monitoring, and operational documentation.

Skills

Databricks
PySpark
ETL/ELT development
Data lake architecture
Data governance
Data quality
Unit testing
Integration testing
Data lineage

Tools

Snowflake
Control-M

Job description

CloudBigD in Austin, TX is seeking 5 Senior Databricks Engineers to modernize data pipelines by migrating legacy Ab Initio ETL to PySpark on Databricks. You will design scalable ETL/ELT processes, refactor monolithic workflows, and integrate Snowflake with robust data quality and governance.

The role covers batch and near real-time processing, data lineage, testing frameworks, and operation runbooks. A strong background in data engineering and enterprise scheduling is required for a successful

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Databricks Engineer - Modern Data Pipelines PySpark
Senior Databricks Engineer - Modern Data Pipelines PySpark

Ethereum Technologies LLC • Austin (TX)

On-site
USD 85,000 - 116,000
Senior Databricks Engineer - PySpark & Data Modernization
Senior Databricks Engineer - PySpark & Data Modernization

Robotic Process Automation LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Databricks Engineer: Data Modernization & PySpark
Senior Databricks Engineer: Data Modernization & PySpark

Quantum Technologies LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Databricks & PySpark Engineer — Data Modernization
Databricks & PySpark Engineer — Data Modernization

Valiant Technologies LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Databricks Engineer
Senior Databricks Engineer

Quantum Technologies LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Databricks Engineer
Senior Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

On-site
USD 85,000 - 116,000
Senior Databricks Engineer
Senior Databricks Engineer

CloudBigD • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Databricks Engineer
Senior Databricks Engineer

Valiant Technologies LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Databricks Engineer
Senior Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

On-site
USD 83,000 - 118,000
Senior Data Engineer - Databricks & Cloud Modernization
Senior Data Engineer - Databricks & Cloud Modernization

Compunnel Inc. • Philadelphia

Hybrid
USD 140,000 - 170,000