Manager-Data Platforms

Buchanan Ingersoll-Rooney; Pittsburgh (Allegheny County); Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

Hybrid Schedule

401K Program

Retirement Savings Program

Generous Paid Time Off

Paid Holidays with floating day

WorkWell wellness program

Gym access

Bright Horizons caregiving assistance

Emergency assistance fund

LinkedIn Learning access

Job summary

Buchanan Ingersoll & Rooney PC seeks a Data Platform Manager in Pittsburgh, PA to design, build, and optimize scalable data platforms on Azure Databricks. You will lead data engineers, collaborate with data scientists, and drive high‑performance data pipelines across the enterprise.

Requirements include 5–7+ years in data engineering/architecture, Azure Databricks expertise, and leadership experience. Hybrid schedule; generous benefits and continued learning opportunities are offered.

Qualifications

• 5-7+ years hands-on data engineering or architecture experience.
• 2-4 years leading a team of data engineers/scientists/analysts is preferred.
• Bachelor's degree in CS, Eng, or related field.
• Certifications in Azure/Data/Databricks are preferred.
• Relational and NoSQL database experience; data modeling and schema design.

Responsibilities

• Lead and mentor a team of data engineers; conduct code reviews and enforce standards.
• Collaborate with stakeholders to translate requirements into technical specs.
• Design, develop, and deploy scalable data pipelines using Azure Databricks; ensure data quality and lineage.
• Optimize pipelines for performance, cost, and scalability in Azure.
• Define end-to-end lakehouse architecture using Delta Lake with Bronze/Silver/Gold layers.
• Implement Unity Catalog governance and RBAC; ensure data security and privacy.
• Oversee batch and streaming ETL/ELT pipelines with PySpark/Scala/SQL.
• Integrate real-time/batch data sources using Apache Kafka and ADF.
• Support ML model deployment with data engineering support.

Skills

Leadership

Mentoring

Data governance

Incident management

PySpark

SQL

Python

Scala

Azure Databricks

Spark

Education

Bachelor's degree in Computer Science, Engineering, or related field

Certifications (Preferred): DP-203, Databricks Data Engineer, Azure Solutions Architect Expert

Tools

Azure Databricks

Azure Data Factory

Azure Synapse

Azure Data Lake Storage

Azure SQL Database

Apache Kafka

Delta Lake

PySpark

SQL

Job description

Buchanan Ingersoll & Rooney is a national law firm with a proven reputation for providing progressive, industry-leading legal, business, regulatory and government relations advice to our regional, national and international clients.

We are searching for a Data Platform Manager for our corporate Pittsburgh, PA location. This role is for a senior technical leader who will be responsible for designing, building, and optimizing scalable enterprise data platforms on the Databricks Data Warehouse on the firm’s Azure Cloud platform. This position combines deep expertise in Databricks with broader knowledge of the Microsoft Azure ecosystem to drive and deliver high-performance data engineering initiatives, data analytics, and data science solutions. This role requires hands-on experience with Azure data services, including Azure Data Lake Storage, Azure SQL Database, Azure Data Factory, Azure Databricks, and Azure Synapse Data Warehouse

The ideal candidate will possess a strong foundation in cloud data platforms and streaming technologies, combined with a leadership mindset to mentor and guide teams in delivering high‑quality solutions. Their role is critical in delivering scalable, robust data solutions that drive actionable insights and support decision‑making.

Hybrid Schedule: 3 days in office, 2 remotely from home

Primary Duties and Responsibilities:

• Lead and mentor a team of data engineers, conducting code reviews and ensuring development standards. Support troubleshooting and incident management for data‑related issues in production.
• Collaborate with business stakeholders, data scientists, and other team members to gather requirements and translate them into technical specifications.
• Lead the design, development and deployment of scalable and high‑performance data pipelines using Azure Databricks; ensuring the data integrity, availability, efficient extraction, transformation, and loading of data from various sources into the firm’s Azure Databricks Data Warehouse.
• Collaborate with data scientists, analysts, and other engineering teams to deliver business‑critical insights. Optimize pipeline performance, cost, and scalability in the Azure cloud environment.
• Define best practices for data ingestion, processing, storage, and governance. Implement data quality checks and validation procedures to ensure the accuracy and integrity of data between various sources, including API’s, databases and streaming platforms
• Collaborate with data scientists and analysts to operationalize and deploy machine learning models.
• Define the end‑to‑end Lakehouse architecture using Delta Lake, implementing medallion architecture (Bronze, Silver, Gold layers) for robust data processing.
• Familiarity with data modeling and schema design principles.
• Oversee the development of robust, scalable batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL and with minimal latency
• Implement data transformations, enrichment, and quality checks using PySpark/Scala within the Databricks environment.
• Integrate real‑time and batch data sources using Apache Kafka and ADF.
• Support large‑scale data pipelines using Apache Spark on Databricks, Kafka, Stelo, and Azure Data Factory (ADF)
• Data Governance & Security:
  • Implement Unity Catalog for unified governance, data security, fine‑grained access control (RBAC), privacy measures, and data lineage tracking.
• Performance Optimization & Tuning:
  • Tune Spark jobs and Databricks clusters to maximize throughput while maintaining cost efficiency through auto‑scaling and cluster policies.
  • Expertise inindexing strategies, query optimization, execution plans, and partitioning/sharding.
• Platform Integration:
  • Orchestrate workflows by integrating Databricks with other Azure services likeAzure Data Factory (ADF),Azure Data Lake Storage (ADLS Gen2), andAzure DevOpsforCI/CDpipelines.

Required Education/Experience:

• 5-7+ years hands‑on data engineering or architecture, with at least 2-4 years specifically focused on Azure Databricks. And Azure cloud technologies.
• 2-5 years experience is preferred in managing a team of data engineers, data scientists and/or analysts.
• Bachelor's degree in Computer Science, Engineering, or a related field.
• Certifications (Preferred): Microsoft Certified: Azure Data Engineer Associate (DP-203), Databricks Certified Data Engineer Professional, orAzure Solutions Architect Expert
• Database Architecture:Proficiency in bothRelational (SQL)andNoSQL(Document, Key-Value, Graph, Columnar) databases. Develop and maintain data models and schemas to support data analysis and reporting requirements
• Distributed Systems:Knowledge of frameworks likeApache Hadoop, Spark, or Presto/Trinofor optimizing and handling massive data volumes and retrieval mechanisms, ensuring the efficient processing of large datasets.
• Storage Optimization:Understanding file formats like Parquet, Avro, or ORC and compression techniques.
• Deep proficiency in programming languages: Python (specifically PySpark), SQL, PowerShell, and Scala.
• Infrastructure: Hands‑on experience with Azure Cloud infrastructure, including Networking (VNETs), Key Vault, and Identity Management. Stays updated with the latest Azure and enterprise cloud data technologies
• Big Data Tools: Deep knowledge ofApache Spark runtime internals,MLflowfor MLOps, and orchestration tools likeAirflow.

Buchanan Ingersoll and Rooney PC offers outstanding benefits that include:

• Hybrid Schedule
• 401K Program
• Retirement Savings Program
• Generous Paid Time Off
• Paid Holidays including a floating holiday
• WorkWell wellness program
• Free use of building gym
• Caregiving assistance with Bright Horizons (child, elder, and pet care!)
• Firm‑wide emergency assistance fund
• Free full access to LinkedIn Learning

We are an Equal Opportunity Employer.

Equal Opportunity Employer/Protected Veterans/Individuals with Disabilities
This employer is required to notify all applicants of their rights pursuant to federal employment laws.For further information, please review the Know Your Rights notice from the Department of Labor.