Azure Databricks Data Platform Lead & Lakehouse Architect

Pittsburgh Tomorrow

Pittsburgh (Allegheny County)

On-site

USD 120,000 - 180,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Pittsburgh Tomorrow is seeking an experienced Data Platform Manager to lead the design, development, and optimization of our enterprise data platforms on Microsoft Azure. You will supervise Databricks architecture, data engineering, and ETL/ELT pipelines while guiding a team of engineers.

The role requires hands-on PySpark, SQL, and Azure data services expertise, plus technical leadership in data governance, security, and scalable analytics.

Qualifications

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • 5–7+ years of hands-on experience in data engineering or data architecture.
  • 2–4 years of direct experience with Azure Databricks and Azure cloud technologies.
  • Strong hands-on experience designing and implementing Azure Databricks Lakehouse architectures.
  • Advanced programming experience with Python, PySpark, SQL, Scala, and PowerShell.
  • Experience developing batch and streaming ETL/ELT data pipelines.
  • Experience with Azure Data Factory, Azure Data Lake Storage, Azure SQL Database, and Azure Synapse.
  • Strong understanding of Apache Spark, Delta Lake, and distributed data processing.
  • Experience with relational SQL and NoSQL databases.
  • Knowledge of data modeling, schema design, and database architecture.
  • Experience with data governance, security, and Unity Catalog.
  • Experience optimizing Spark workloads, Databricks clusters, and cloud infrastructure.
  • Knowledge of Azure networking, VNETs, Key Vault, and identity management.
  • Experience with data orchestration, automation, and CI/CD processes.

Responsibilities

  • Lead, mentor, and provide technical direction to data engineers, including code reviews and production troubleshooting.
  • Design, build, and maintain scalable enterprise data platforms using Azure Databricks and Microsoft Azure.
  • Architect and implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
  • Oversee batch and real-time ETL/ELT pipelines using PySpark, Scala, SQL, and Spark.
  • Integrate data from APIs, relational databases, streaming platforms, and other sources.
  • Develop and optimize data ingestion, transformation, enrichment, and validation processes.
  • Integrate Databricks with Azure Data Factory, Data Lake Storage Gen2, SQL Database, and Synapse.
  • Implement Unity Catalog for data governance, access controls, data lineage, privacy, and security.
  • Optimize Databricks clusters, Spark jobs, and cloud resource utilization for performance and cost.
  • Support streaming data integrations using Apache Kafka and other distributed processing technologies.
  • Implement CI/CD pipelines and deployment automation using Azure DevOps.
  • Collaborate with data scientists and analysts to support ML models, analytics, and BI initiatives.
  • Establish data engineering standards, quality controls, monitoring, and operational best practices.
  • Partner with business stakeholders to translate data requirements into scalable technical solutions.
  • Support cloud infrastructure security, networking, identity management, and access controls.

Skills

Technical leadership
Data engineering
Cloud architecture
Python
SQL
Stakeholder collaboration
Problem solving

Education

Bachelor's degree in CS/Engineering or related field

Tools

Azure Databricks
Azure Data Factory
Delta Lake
Azure Synapse
Azure Data Lake Storage
Azure DevOps
Kafka
Spark

Job description

Pittsburgh Tomorrow is seeking an experienced Data Platform Manager to lead the design, development, and optimization of our enterprise data platforms on Microsoft Azure. You will supervise Databricks architecture, data engineering, and ETL/ELT pipelines while guiding a team of engineers.

The role requires hands-on PySpark, SQL, and Azure data services expertise, plus technical leadership in data governance, security, and scalable analytics.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer - Azure Databricks & Lakehouse
Lead Data Engineer - Azure Databricks & Lakehouse

Life in Pittsburgh • Pittsburgh

On-site
USD 140,000 - 190,000
Azure Databricks Data Platform Lead
Azure Databricks Data Platform Lead

Buchanan Ingersoll-Rooney • Pittsburgh

Hybrid
USD 150,000 - 210,000
Hybrid Schedule
401K Program
Retirement Savings Program
+7
Azure Databricks Data Platform Leader (Hybrid)
Azure Databricks Data Platform Leader (Hybrid)

Buchanan Ingersoll & Rooney Pc • Pittsburgh

Hybrid
USD 140,000 - 180,000
Hybrid Schedule
Medical, Dental, Vision insurance
401K Program
+2
Azure Databricks Tech Lead - Data Platform
Azure Databricks Tech Lead - Data Platform

CloudBigD • Charlotte (NC)

On-site
USD 130,000 - 170,000
Manager Data Platforms
Manager Data Platforms

Pittsburgh Tomorrow • Pittsburgh

On-site
USD 120,000 - 180,000
Senior Azure Databricks Platform Architect (Remote)
Senior Azure Databricks Platform Architect (Remote)

Lumenalta • United States

Remote
USD 80,000 - 150,000
Health insurance
401K Contribution
Professional development fund
+1
Senior Azure Databricks Lakehouse Architect
Senior Azure Databricks Lakehouse Architect

Programmers • Minneapolis (MN)

On-site
USD 120,000 - 170,000
Remote Data Platform Leader (Azure & Databricks)
Remote Data Platform Leader (Azure & Databricks)

World Business Lenders, LLC • United States

Remote
USD 180,000 - 240,000
Azure Data Platform Lead: Lakehouse & ML Enablement
Azure Data Platform Lead: Lakehouse & ML Enablement

Confidentiel • Pittsburgh

Hybrid
USD 140,000 - 190,000
Data Platform Lead — Azure & Databricks (Hybrid)
Data Platform Lead — Azure & Databricks (Hybrid)

Pennant Solutions Group • Richmond (VA)

Hybrid
USD 120,000 - 180,000