Principal Data Engineer

Pittsburgh Tomorrow

Pittsburgh (Allegheny County)

On-site

USD 140,000 - 190,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Pittsburgh Tomorrow is seeking an experienced Data Engineering Manager to lead the development of scalable data platforms on Microsoft Azure. You will design, build, and optimize data solutions, mentor a talented team, and ensure robust data governance and security across enterprise systems.

The role emphasizes hands-on leadership in Azure Databricks, PySpark, Python, SQL, and Lakehouse architectures while collaborating with cross-functional partners to enable data-driven decision making and

Qualifications

  • 5-7+ years of experience in data engineering, data architecture, or related technical roles.
  • 2-4 years hands-on experience with Azure Databricks.
  • Strong experience with PySpark, Python, SQL, and Apache Spark.
  • Hands-on experience with Delta Lake, Lakehouse Architecture, Medallion Architecture.
  • Experience building and managing batch and streaming data pipelines using Azure Data Factory.
  • Strong understanding of data governance, pipeline optimization, and Azure cloud infrastructure.
  • Experience designing and implementing production-level data engineering solutions.

Responsibilities

  • Lead design, development, and implementation of scalable enterprise data platforms using Azure Databricks.
  • Build, maintain, and optimize batch and streaming data pipelines using Azure Data Factory and other Azure data services.
  • Design and implement Lakehouse Architecture, Delta Lake, and Medallion Architecture.
  • Develop and optimize data processing solutions using PySpark, Python, SQL, and Apache Spark.
  • Improve data pipeline performance, scalability, reliability, and efficiency.
  • Implement data governance, security, and quality standards across enterprise data platforms.
  • Provide technical leadership, mentorship, and guidance to data engineering team members.
  • Collaborate with cross-functional teams to deliver reliable data solutions that support business and analytics needs.

Skills

Data engineering
Data architecture
Azure Databricks
PySpark
Python
SQL
Apache Spark
Lakehouse Architecture
Medallion Architecture
Azure Data Factory

Tools

Delta Lake
Unity Catalog
Kafka
MLflow
Apache Airflow
Azure DevOps
CI/CD

Job description

About the Opportunity

We are seeking an experienced Data Engineering Manager to lead and support the development of modern, scalable data platforms. This individual will play a key role in designing, building, and optimizing enterprise-level data solutions using Microsoft Azure and Azure Databricks. The ideal candidate will bring strong hands‑on data engineering experience, particularly with Azure Databricks, PySpark, Python, SQL, and Lakehouse Architecture. This role is suited for a technical leader who enjoys remaining involved in development while guiding and mentoring data engineering teams. We are also open to experienced Lead Data Engineers, Principal Data Engineers, and Databricks Technical Leads who have strong technical expertise and leadership capabilities.

Key Responsibilities
  • Lead the design, development, and implementation of scalable enterprise data platforms using Azure Databricks.
  • Build, maintain, and optimize batch and streaming data pipelines using Azure Data Factory and other Azure data services.
  • Design and implement Lakehouse Architecture, Delta Lake, and Medallion Architecture.
  • Develop and optimize data processing solutions using PySpark, Python, SQL, and Apache Spark.
  • Improve data pipeline performance, scalability, reliability, and efficiency.
  • Implement data governance, security, and quality standards across enterprise data platforms.
  • Provide technical leadership, mentorship, and guidance to data engineering team members.
  • Collaborate with cross-functional teams to deliver reliable data solutions that support business and analytics needs.
Must-Have Qualifications
  • 5-7+ years of experience in data engineering, data architecture, or related technical roles.
  • 2-4 years of hands‑on experience working with Azure Databricks.
  • Strong experience with PySpark, Python, SQL, and Apache Spark.
  • Hands‑on experience with Delta Lake, Lakehouse Architecture, and Medallion Architecture.
  • Experience building and managing batch and streaming data pipelines using Azure Data Factory and Azure data services.
  • Strong understanding of data governance, pipeline optimization, and Azure cloud infrastructure.
  • Experience designing and implementing production‑level data engineering solutions.
Nice-to-Have Qualifications
  • Experience managing, leading, or mentoring data engineering teams.
  • Knowledge of Scala and PowerShell.
  • Experience with Unity Catalog, Kafka, MLflow, and Apache Airflow.
  • Familiarity with Azure DevOps and CI/CD practices.
  • Relevant certifications, including Azure Data Engineer, Databricks Data Engineer, or Azure Solutions Architect.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Manager Data Platforms
Manager Data Platforms

Pittsburgh Tomorrow • Pittsburgh

On-site
USD 120,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Confidentiel • Pittsburgh

Hybrid
USD 150,000 - 190,000
Director Data Engineering
Director Data Engineering

Straive • United States

Remote
USD 180,000 - 240,000
Sr. Data Engineer, Data Platform
Sr. Data Engineer, Data Platform

Mirion Technologies • United States

On-site
USD 120,000 - 160,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 180,000 - 240,000
Cloud Platform Manager
Cloud Platform Manager

Confidentiel • Pittsburgh

Hybrid
USD 140,000 - 190,000
Data Engineer
Data Engineer

Compunnel Inc. • San Jose (CA)

On-site
USD 120,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

EXL Talent Acquisition Team • New York (NY)

On-site
USD 170,000 - 230,000
Data Engineering Lead
Data Engineering Lead

Elevanta Partners • New York (NY)

On-site
USD 150,000 - 200,000
Senior Data Engineer on-site)
Senior Data Engineer on-site)

Ziosk • Dallas (TX)

On-site
USD 140,000 - 190,000