Manager Data Platforms

Pittsburgh Tomorrow

Pittsburgh (Allegheny County)

On-site

USD 120,000 - 180,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Pittsburgh Tomorrow is seeking an experienced Data Platform Manager to lead the design, development, and optimization of our enterprise data platforms on Microsoft Azure. You will supervise Databricks architecture, data engineering, and ETL/ELT pipelines while guiding a team of engineers.

The role requires hands-on PySpark, SQL, and Azure data services expertise, plus technical leadership in data governance, security, and scalable analytics.

Qualifications

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • 5–7+ years of hands-on experience in data engineering or data architecture.
  • 2–4 years of direct experience with Azure Databricks and Azure cloud technologies.
  • Strong hands-on experience designing and implementing Azure Databricks Lakehouse architectures.
  • Advanced programming experience with Python, PySpark, SQL, Scala, and PowerShell.
  • Experience developing batch and streaming ETL/ELT data pipelines.
  • Experience with Azure Data Factory, Azure Data Lake Storage, Azure SQL Database, and Azure Synapse.
  • Strong understanding of Apache Spark, Delta Lake, and distributed data processing.
  • Experience with relational SQL and NoSQL databases.
  • Knowledge of data modeling, schema design, and database architecture.
  • Experience with data governance, security, and Unity Catalog.
  • Experience optimizing Spark workloads, Databricks clusters, and cloud infrastructure.
  • Knowledge of Azure networking, VNETs, Key Vault, and identity management.
  • Experience with data orchestration, automation, and CI/CD processes.

Responsibilities

  • Lead, mentor, and provide technical direction to data engineers, including code reviews and production troubleshooting.
  • Design, build, and maintain scalable enterprise data platforms using Azure Databricks and Microsoft Azure.
  • Architect and implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
  • Oversee batch and real-time ETL/ELT pipelines using PySpark, Scala, SQL, and Spark.
  • Integrate data from APIs, relational databases, streaming platforms, and other sources.
  • Develop and optimize data ingestion, transformation, enrichment, and validation processes.
  • Integrate Databricks with Azure Data Factory, Data Lake Storage Gen2, SQL Database, and Synapse.
  • Implement Unity Catalog for data governance, access controls, data lineage, privacy, and security.
  • Optimize Databricks clusters, Spark jobs, and cloud resource utilization for performance and cost.
  • Support streaming data integrations using Apache Kafka and other distributed processing technologies.
  • Implement CI/CD pipelines and deployment automation using Azure DevOps.
  • Collaborate with data scientists and analysts to support ML models, analytics, and BI initiatives.
  • Establish data engineering standards, quality controls, monitoring, and operational best practices.
  • Partner with business stakeholders to translate data requirements into scalable technical solutions.
  • Support cloud infrastructure security, networking, identity management, and access controls.

Skills

Technical leadership
Data engineering
Cloud architecture
Python
SQL
Stakeholder collaboration
Problem solving

Education

Bachelor's degree in CS/Engineering or related field

Tools

Azure Databricks
Azure Data Factory
Delta Lake
Azure Synapse
Azure Data Lake Storage
Azure DevOps
Kafka
Spark

Job description

Seeking an experienced Data Platform Manager to lead the design, development, and optimization of enterprise data platforms within Microsoft Azure. This position will oversee Azure Databricks architecture, data engineering, ETL/ELT pipelines, data governance, and cloud infrastructure integration. The ideal candidate combines hands-on expertise in Databricks, PySpark, SQL, and Azure data services with technical leadership experience managing data engineering teams and delivering scalable analytics solutions.

Key Responsibilities
  • Lead, mentor, and provide technical direction to data engineers, including code reviews, development standards, and production troubleshooting.

  • Design, build, and maintain scalable enterprise data platforms using Azure Databricks and Microsoft Azure.

  • Architect and implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers).

  • Oversee the development of batch and real-time ETL/ELT pipelines using PySpark, Scala, SQL, and Apache Spark.

  • Integrate data from APIs, relational databases, streaming platforms, and other enterprise sources.

  • Develop and optimize data ingestion, transformation, enrichment, and validation processes.

  • Integrate Azure Databricks with Azure Data Factory, Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Synapse.

  • Implement Unity Catalog for data governance, access controls, data lineage, privacy, and security.

  • Optimize Databricks clusters, Spark jobs, query execution, and cloud resource utilization for performance and cost efficiency.

  • Support streaming data integrations using Apache Kafka and other distributed processing technologies.

  • Implement CI/CD pipelines and deployment automation using Azure DevOps.

  • Collaborate with data scientists and analysts to support machine learning models, analytics, and business intelligence initiatives.

  • Establish data engineering standards, quality controls, monitoring, and operational best practices.

  • Partner with business stakeholders to translate data requirements into scalable technical solutions.

  • Support cloud infrastructure security, networking, identity management, and access controls.

Minimum Education & Experience
  • Bachelor's degree in Computer Science, Engineering, or a related field.

  • Five to seven or more years of hands-on experience in data engineering or data architecture.

  • Two to four years of direct experience with Azure Databricks and Azure cloud technologies.

  • Strong hands-on experience designing and implementing Azure Databricks Lakehouse architectures.

  • Advanced programming experience with Python, PySpark, SQL, Scala, and PowerShell.

  • Experience developing batch and streaming ETL/ELT data pipelines.

  • Experience with Azure Data Factory, Azure Data Lake Storage, Azure SQL Database, and Azure Synapse.

  • Strong understanding of Apache Spark, Delta Lake, and distributed data processing.

  • Experience with relational SQL and NoSQL databases.

  • Knowledge of data modeling, schema design, and database architecture.

  • Experience with data governance, security, and Unity Catalog.

  • Experience optimizing Spark workloads, Databricks clusters, and cloud infrastructure.

  • Knowledge of Azure networking, VNETs, Key Vault, and identity management.

  • Experience with data orchestration, automation, and CI/CD processes.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Confidentiel • Pittsburgh

Hybrid
USD 150,000 - 190,000
Cloud Platform Manager
Cloud Platform Manager

Confidentiel • Pittsburgh

Hybrid
USD 140,000 - 190,000
Principal Data Engineer
Principal Data Engineer

Pittsburgh Tomorrow • Pittsburgh

On-site
USD 140,000 - 190,000
Director Data Engineering
Director Data Engineering

Straive • United States

Remote
USD 180,000 - 240,000
Senior Azure Databricks Engineer (ESS-424)
Senior Azure Databricks Engineer (ESS-424)

Environmental and Safety Solutions, Inc. • Glendale (AZ)

On-site
USD 140,000 - 180,000
Sr. Data Engineer, Data Platform
Sr. Data Engineer, Data Platform

Mirion Technologies • United States

On-site
USD 120,000 - 160,000
Databricks Data Architect
Databricks Data Architect

NewVision • San Francisco (CA)

Hybrid
USD 140,000 - 190,000
Hybrid work model
Databricks Data Architect
Databricks Data Architect

NewVision Software • United States

Remote
USD 120,000 - 180,000
Data Platform Administration Architect
Data Platform Administration Architect

Eklm • United States

Remote
USD 150,000 - 190,000
Manager-Data Platforms
Manager-Data Platforms

Buchanan Ingersoll & Rooney Pc • Pittsburgh

Hybrid
USD 140,000 - 180,000
Hybrid Schedule
Medical, Dental, Vision insurance
401K Program
+2