Senior Azure Data Engineer: Pipelines, Databricks & Big Data

CloudBigD

Houston (TX)

On-site

USD 120,000 - 160,000

Full time

Just now
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

CloudBigD in Houston, TX seeks a senior Data Engineer to develop Azure Data Factory pipelines, Databricks notebooks, and Delta Lake solutions, building modular data workflows with Python or Scala and robust unit tests.

You will tune SQL for large datasets, manage ADLS Gen2 storage, partitioning and OPTIMIZE/VACUUM, and support Git-based CI/CD across Azure environments, collaborating with stakeholders through testing and delivery cycles.

Qualifications

  • 8-10 years of data engineering experience
  • At least 3 years of Azure experience
  • Advanced SQL
  • Python or Scala
  • Experience with large datasets and data pipeline optimization

Responsibilities

  • Develop Azure Data Factory pipelines, data flows, triggers, integration runtimes, and parameterized workflows.
  • Build and optimize Azure Databricks notebooks, jobs, workflows, and Delta Lake solutions.
  • Develop modular data engineering code using Python or Scala with unit testing practices.
  • Write and tune advanced SQL queries using CTEs, window functions, and large datasets.
  • Manage data storage and optimization using ADLS Gen2, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.
  • Support Git-based CI/CD and promotion across Azure Data Factory and Databricks environments.
  • Collaborate with stakeholders and participate in testing and delivery activities.

Skills

Azure Data Factory pipelines
Azure Databricks
Python/Scala
Advanced SQL
Data pipeline optimization
Large datasets handling

Job description

CloudBigD in Houston, TX seeks a senior Data Engineer to develop Azure Data Factory pipelines, Databricks notebooks, and Delta Lake solutions, building modular data workflows with Python or Scala and robust unit tests.

You will tune SQL for large datasets, manage ADLS Gen2 storage, partitioning and OPTIMIZE/VACUUM, and support Git-based CI/CD across Azure environments, collaborating with stakeholders through testing and delivery cycles.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Azure Data Engineer: Pipelines & Databricks
Senior Azure Data Engineer: Pipelines & Databricks

Quantum Technologies LLC • Houston (TX)

On-site
USD 120,000 - 180,000
Senior Azure Data Engineer: Pipelines, Databricks & SQL
Senior Azure Data Engineer: Pipelines, Databricks & SQL

Ethereum Technologies LLC • Houston (TX)

On-site
USD 120,000 - 180,000
Senior Azure Data Engineer: Pipelines, Databricks & SQL
Senior Azure Data Engineer: Pipelines, Databricks & SQL

Valiant Technologies LLC • Houston (TX)

On-site
USD 120,000 - 170,000
Azure Data Engineer Lead: Pipelines, Databricks & Delta Lake
Azure Data Engineer Lead: Pipelines, Databricks & Delta Lake

Robotic Process Automation LLC • Houston (TX)

On-site
USD 120,000 - 170,000
Senior Azure Data Engineer — Pipelines & Databricks
Senior Azure Data Engineer — Pipelines & Databricks

Speridian • Austin (TX)

On-site
USD 140,000 - 190,000
Azure Data Engineer: Build Scalable Pipelines & Lakehouse
Azure Data Engineer: Build Scalable Pipelines & Lakehouse

CloudBigD • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Databricks Data Engineer - Lakehouse & Pipelines
Azure Databricks Data Engineer - Lakehouse & Pipelines

Ethereum Technologies LLC • Dallas (TX)

On-site
USD 120,000 - 160,000
Senior Azure Data Engineer - Pipelines & Databricks
Senior Azure Data Engineer - Pipelines & Databricks

Insight Global (United States) • Portland (OR)

On-site
USD 120,000 - 160,000
Senior Azure Databricks Data Engineer: Delta Lake Pipelines
Senior Azure Databricks Data Engineer: Delta Lake Pipelines

EXL • Detroit (MI)

On-site
USD 120,000 - 190,000
Azure Data Engineer: Spark, Delta Lake & Pipelines
Azure Data Engineer: Spark, Delta Lake & Pipelines

TechDigital Group • Pleasanton (CA)

On-site
USD 120,000 - 170,000