Senior Azure Data Engineer: Pipelines & Databricks

Quantum Technologies LLC

Houston (TX)

On-site

USD 120,000 - 180,000

Full time

22 hours ago
Be an early applicant

Get a reply from this recruiter

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Quantum Technologies LLC in Houston, TX is seeking an experienced Data Engineer to design and optimize data pipelines using Azure Data Factory and Databricks. You will develop modular data engineering code in Python or Scala, implement advanced SQL queries, and manage storage optimizations with ADLS Gen2 and Delta Lake.

Collaboration with stakeholders and CI/CD practices are essential. Ideal candidates have 8-10 years in data engineering, at least 3 years on Azure, and strong skills in SQL,

Qualifications

  • 8-10 years of data engineering experience.
  • At least 3 years of Azure experience.
  • Advanced SQL.
  • Python or Scala.
  • Experience with large datasets and data pipeline optimization.

Responsibilities

  • Develop Azure Data Factory pipelines, data flows, triggers, integration runtimes, and parameterized workflows.
  • Build and optimize Azure Databricks notebooks, jobs, workflows, and Delta Lake solutions.
  • Develop modular data engineering code using Python or Scala with unit testing practices.
  • Write and tune advanced SQL queries using CTEs, window functions, and large datasets.
  • Manage data storage and optimization using ADLS Gen2, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.
  • Support Git-based CI/CD and promotion across Azure Data Factory and Databricks environments.
  • Collaborate with stakeholders and participate in testing and delivery activities.

Skills

Azure Data Factory pipelines
Azure Databricks
Python
Scala
Advanced SQL
Large datasets
Data pipeline optimization
Git CI/CD

Tools

ADLS Gen2
Parquet
Delta Lake
Azure Data Factory
Databricks notebooks

Job description

Quantum Technologies LLC in Houston, TX is seeking an experienced Data Engineer to design and optimize data pipelines using Azure Data Factory and Databricks. You will develop modular data engineering code in Python or Scala, implement advanced SQL queries, and manage storage optimizations with ADLS Gen2 and Delta Lake.

Collaboration with stakeholders and CI/CD practices are essential. Ideal candidates have 8-10 years in data engineering, at least 3 years on Azure, and strong skills in SQL,

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer — Databricks & ADF Pipelines
Data Engineer — Databricks & ADF Pipelines

Quantum Technologies. LLC • Atlanta (GA)

On-site
USD 90,000 - 150,000
Senior Azure Data Engineer: Pipelines, Databricks & SQL
Senior Azure Data Engineer: Pipelines, Databricks & SQL

Valiant Technologies LLC • Houston (TX)

On-site
USD 120,000 - 170,000
Senior Azure Data Engineer: Pipelines, Databricks & SQL
Senior Azure Data Engineer: Pipelines, Databricks & SQL

Ethereum Technologies LLC • Houston (TX)

On-site
USD 120,000 - 180,000
Azure Data Engineer – ETL, Databricks & Data Lake Architect
Azure Data Engineer – ETL, Databricks & Data Lake Architect

Quantum Technologies. LLC • Atlanta (GA)

On-site
USD 120,000 - 150,000
Azure Data Engineer — Lead Data Pipelines & Databricks
Azure Data Engineer — Lead Data Pipelines & Databricks

Quantum Technologies LLC • Pennsylvania

On-site
USD 91,000 - 124,000
Azure Data Engineer — Pipelines, Databricks & Power BI
Azure Data Engineer — Pipelines, Databricks & Power BI

Quantum Technologies. LLC • Columbia Township (MO)

On-site
USD 90,000 - 130,000
Azure Data Engineer: ETL, Databricks & Synapse
Azure Data Engineer: ETL, Databricks & Synapse

Quantum Technologies. LLC • Austin (TX)

On-site
USD 120,000 - 180,000
Senior Azure Data Engineer — Pipelines & Databricks
Senior Azure Data Engineer — Pipelines & Databricks

Speridian • Austin (TX)

On-site
USD 140,000 - 190,000
Data Engineer - Databricks & Azure Data Factory
Data Engineer - Databricks & Azure Data Factory

Quantum Technologies LLC • Plano (TX)

On-site
USD 98,000 - 139,000
Senior Azure Data Architect — Databricks & Power BI
Senior Azure Data Architect — Databricks & Power BI

Quantum Technologies LLC • Minnesota

On-site
USD 110,000 - 150,000