Databricks & PySpark Engineer — Data Modernization

Valiant Technologies LLC

Austin (TX)

Presencial

USD 83 000 - 118 000

Tempo integral

Há 5 dias
Torna-te num dos primeiros candidatos

Uma candidatura completa num minuto

Currículo e carta de apresentação personalizados, prontos a enviar.

Ultrapassa os filtros ATS

Resumo da oferta

Valiant Technologies LLC in Austin, TX, is seeking Senior Databricks Engineers to support a large-scale data modernization initiative. You will migrate legacy Ab Initio ETL workflows into modular PySpark pipelines on Databricks, delivering scalable data solutions.

The role requires hands-on Databricks and PySpark experience, ETL modernization, Snowflake integration, and governance. This contract spans 12 months with potential extension and uses enterprise scheduling tools like Control-M.

Qualificações

  • Strong hands-on Databricks and PySpark experience.
  • Experience migrating Ab Initio ETL workflows."
  • ETL/ELT development and data lake architecture experience.
  • Snowflake data integration experience.
  • Data lineage, governance, and data quality.

Responsabilidades

  • Analyze and migrate legacy Ab Initio workflows into PySpark-based ETL pipelines.
  • Design, develop, and optimize scalable data pipelines using Databricks and PySpark.
  • Refactor monolithic ETL processes into modular, reusable components.
  • Build ETL/ELT pipelines integrating Snowflake and other enterprise data sources.
  • Support batch and near real-time data processing.
  • Create end-to-end data lineage, data flow diagrams, and identify opportunities for process consolidation.
  • Implement data quality, governance, and validation standards.
  • Develop unit, integration, and reconciliation testing frameworks.
  • Perform dual-run comparisons between legacy and modernized pipelines.
  • Support UAT, pre-production validation, migration cutover, and legacy workflow decommissioning.
  • Optimize job scheduling using tools such as Control-M.
  • Prepare runbooks, monitoring solutions, and operational documentation.

Conhecimentos

Databricks
PySpark
ETL development
Snowflake
Data governance
Unit testing
Control-M

Ferramentas

Control-M

Descrição da oferta de emprego

Valiant Technologies LLC in Austin, TX, is seeking Senior Databricks Engineers to support a large-scale data modernization initiative. You will migrate legacy Ab Initio ETL workflows into modular PySpark pipelines on Databricks, delivering scalable data solutions.

The role requires hands-on Databricks and PySpark experience, ETL modernization, Snowflake integration, and governance. This contract spans 12 months with potential extension and uses enterprise scheduling tools like Control-M.

Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

Senior Databricks Engineer - Data Modernization & PySpark
Senior Databricks Engineer - Data Modernization & PySpark

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer - Modern Data Pipelines PySpark
Senior Databricks Engineer - Modern Data Pipelines PySpark

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000
Senior Databricks Engineer: Data Modernization & PySpark
Senior Databricks Engineer: Data Modernization & PySpark

Quantum Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer - PySpark & Data Modernization
Senior Databricks Engineer - PySpark & Data Modernization

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Robotic Process Automation LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Valiant Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Ethereum Technologies LLC • Austin (TX)

Presencial
USD 85 000 - 116 000
Senior Databricks Engineer
Senior Databricks Engineer

CloudBigD • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Engineer
Senior Databricks Engineer

Quantum Technologies LLC • Austin (TX)

Presencial
USD 83 000 - 118 000
Senior Databricks Data Engineer: Spark, ETL & AWS
Senior Databricks Data Engineer: Spark, ETL & AWS

Valiant Technologies LLC • Dallas (TX)

Presencial
USD 120 000 - 190 000