Lead Data Engineer - Azure Databricks & Lakehouse

Life in Pittsburgh

Pittsburgh (Allegheny County)

On-site

USD 140,000 - 190,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Pittsburgh Tomorrow is seeking an experienced Data Engineering Manager to lead the development of scalable data platforms on Microsoft Azure. You will design, build, and optimize data solutions, mentor a talented team, and ensure robust data governance and security across enterprise systems.

The role emphasizes hands-on leadership in Azure Databricks, PySpark, Python, SQL, and Lakehouse architectures while collaborating with cross-functional partners to enable data-driven decision making and

Qualifications

  • 5-7+ years of experience in data engineering, data architecture, or related technical roles.
  • 2-4 years hands-on experience with Azure Databricks.
  • Strong experience with PySpark, Python, SQL, and Apache Spark.
  • Hands-on experience with Delta Lake, Lakehouse Architecture, Medallion Architecture.
  • Experience building and managing batch and streaming data pipelines using Azure Data Factory.
  • Strong understanding of data governance, pipeline optimization, and Azure cloud infrastructure.
  • Experience designing and implementing production-level data engineering solutions.

Responsibilities

  • Lead design, development, and implementation of scalable enterprise data platforms using Azure Databricks.
  • Build, maintain, and optimize batch and streaming data pipelines using Azure Data Factory and other Azure data services.
  • Design and implement Lakehouse Architecture, Delta Lake, and Medallion Architecture.
  • Develop and optimize data processing solutions using PySpark, Python, SQL, and Apache Spark.
  • Improve data pipeline performance, scalability, reliability, and efficiency.
  • Implement data governance, security, and quality standards across enterprise data platforms.
  • Provide technical leadership, mentorship, and guidance to data engineering team members.
  • Collaborate with cross-functional teams to deliver reliable data solutions that support business and analytics needs.

Skills

Data engineering
Data architecture
Azure Databricks
PySpark
Python
SQL
Apache Spark
Lakehouse Architecture
Medallion Architecture
Azure Data Factory

Tools

Delta Lake
Unity Catalog
Kafka
MLflow
Apache Airflow
Azure DevOps
CI/CD

Job description

Pittsburgh Tomorrow is seeking an experienced Data Engineering Manager to lead the development of scalable data platforms on Microsoft Azure. You will design, build, and optimize data solutions, mentor a talented team, and ensure robust data governance and security across enterprise systems.

The role emphasizes hands-on leadership in Azure Databricks, PySpark, Python, SQL, and Lakehouse architectures while collaborating with cross-functional partners to enable data-driven decision making and

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Azure Databricks Data Platform Lead & Lakehouse Architect
Azure Databricks Data Platform Lead & Lakehouse Architect

Pittsburgh Tomorrow • Pittsburgh

On-site
USD 120,000 - 180,000
Lead Data Engineer - Azure Databricks & Lakehouse
Lead Data Engineer - Azure Databricks & Lakehouse

EPAM Systems • United States

Remote
USD 140,000 - 170,000
Azure Databricks Data Platform Lead
Azure Databricks Data Platform Lead

Buchanan Ingersoll-Rooney • Pittsburgh

Hybrid
USD 150,000 - 210,000
Hybrid Schedule
401K Program
Retirement Savings Program
+7
Data & Analytics Engineer: Cloud Lakehouse & AI
Data & Analytics Engineer: Cloud Lakehouse & AI

eNGINE • Pittsburgh

On-site
USD 80,000 - 110,000
401(k)
401(k) matching
Dental insurance
+4
Lead Data Engineer — Azure, Databricks & Streaming
Lead Data Engineer — Azure, Databricks & Streaming

8451 • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Senior Databricks Engineer - Azure Lakehouse, Remote
Senior Databricks Engineer - Azure Lakehouse, Remote

DataArt • United States

Hybrid
USD 120,000 - 180,000
Vacation days up to 26 per year
Health and life insurance
MyBenefit platform with Multisport
+6
Lead Data Architect & Engineer - Databricks Lakehouse
Lead Data Architect & Engineer - Databricks Lakehouse

CloudBigD • Atlanta (GA)

On-site
USD 89,000 - 126,000
Lead Data Engineer: Databricks & Azure Streaming Architect
Lead Data Engineer: Databricks & Azure Streaming Architect

84.51˚ • Cincinnati (OH)

On-site
USD 125,000 - 207,000
Health insurance
401(k) with matching
Paid time off
Data Engineering Lead: Cloud Lakehouse & Azure Databricks
Data Engineering Lead: Cloud Lakehouse & Azure Databricks

Tiger Analytics • United States

Remote
USD 170,000 - 210,000
Collaborative work environment
Global client projects
Career development and learning
Data Platform Lead — Azure & Databricks (Hybrid)
Data Platform Lead — Azure & Databricks (Hybrid)

Pennant Solutions Group • Richmond (VA)

Hybrid
USD 120,000 - 180,000