Lead Azure Data Engineer

Quantum Technologies. LLC

Houston (TX)

On-site

USD 120,000 - 180,000

Full time

1 hour ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Quantum Technologies LLC in Houston, TX is seeking an experienced Data Engineer to design and optimize data pipelines using Azure Data Factory and Databricks. You will develop modular data engineering code in Python or Scala, implement advanced SQL queries, and manage storage optimizations with ADLS Gen2 and Delta Lake.

Collaboration with stakeholders and CI/CD practices are essential. Ideal candidates have 8-10 years in data engineering, at least 3 years on Azure, and strong skills in SQL,

Qualifications

  • 8-10 years of data engineering experience.
  • At least 3 years of Azure experience.
  • Advanced SQL.
  • Python or Scala.
  • Experience with large datasets and data pipeline optimization.

Responsibilities

  • Develop Azure Data Factory pipelines, data flows, triggers, integration runtimes, and parameterized workflows.
  • Build and optimize Azure Databricks notebooks, jobs, workflows, and Delta Lake solutions.
  • Develop modular data engineering code using Python or Scala with unit testing practices.
  • Write and tune advanced SQL queries using CTEs, window functions, and large datasets.
  • Manage data storage and optimization using ADLS Gen2, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.
  • Support Git-based CI/CD and promotion across Azure Data Factory and Databricks environments.
  • Collaborate with stakeholders and participate in testing and delivery activities.

Skills

Azure Data Factory pipelines
Azure Databricks
Python
Scala
Advanced SQL
Large datasets
Data pipeline optimization
Git CI/CD

Tools

ADLS Gen2
Parquet
Delta Lake
Azure Data Factory
Databricks notebooks

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-1B, H4-EAD, Other

Job Description:

Develop Azure Data Factory pipelines, data flows, triggers, integration runtimes, and parameterized workflows.

Build and optimize Azure Databricks notebooks, jobs, workflows, and Delta Lake solutions.

Develop modular data engineering code using Python or Scala with unit testing practices.

Write and tune advanced SQL queries using CTEs, window functions, and large datasets.

Manage data storage and optimization using ADLS Gen2, Parquet, partitioning, Z-Ordering, OPTIMIZE, and VACUUM.

Support Git-based CI/CD and promotion across Azure Data Factory and Databricks environments.

Collaborate with stakeholders and participate in testing and delivery activities.

Required Skills & Experience

8-10 years of data engineering experience

At least 3 years of Azure experience

Advanced SQL

Python or Scala

Experience with large datasets and data pipeline optimization

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Azure Data Engineer
Lead Azure Data Engineer

Valiant Technologies LLC • Houston (TX)

On-site
USD 120,000 - 170,000
Lead Azure Data Engineer
Lead Azure Data Engineer

Ethereum Technologies • Houston (TX)

On-site
USD 120,000 - 180,000
Lead Azure Data Engineer
Lead Azure Data Engineer

Robotic Technologies Automation • Houston (TX)

On-site
USD 120,000 - 170,000
Azure Data Engineer
Azure Data Engineer

Ethereum Technologies • Raleigh (NC)

On-site
USD 120,000 - 150,000
None
Azure Data Engineer
Azure Data Engineer

Valiant Technologies LLC • Raleigh (NC)

On-site
USD 110,000 - 150,000
Azure Data Engineer
Azure Data Engineer

Quantum Technologies. LLC • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Robotic Technologies Automation • Raleigh (NC)

On-site
USD 110,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Valiant Technologies LLC • Atlanta (GA)

On-site
USD 110,000 - 160,000
Azure Data Engineer
Azure Data Engineer

Quantum Technologies LLC • Atlanta (GA)

On-site
USD 120,000 - 150,000
Azure Data Engineer
Azure Data Engineer

Ethereum Technologies LLC • Atlanta (GA)

On-site
USD 90,000 - 130,000