Senior Data Engineer

Confidentiel; Pittsburgh (Allegheny County); Hybrid; USD 150,000 - 190,000; Full time

Job summary

Confidentiel seeks a Cloud Platform Manager to lead enterprise data platforms on Azure, architect Lakehouse solutions, optimize Spark workloads, and govern scalable pipelines for analytics and ML.

You will mentor data engineers, collaborate with stakeholders, and ensure security, reliability, and cost efficiency across hybrid cloud environments.

Qualifications

• Bachelor's degree in a technical field.
• 5–7 years hands-on experience in data engineering/cloud architecture.
• 2–4 years direct experience with Azure Databricks/Azure data technologies.
• Experience leading technical teams of data engineers, analysts or data scientists.
• Strong background in SQL, Python/PySpark, Scala, and data modeling.
• Experience with modern cloud-based data platforms and streaming architectures.

Responsibilities

• Lead and mentor data engineers, analysts, and professionals while promoting standards/best practices.
• Design, develop, and optimize scalable data platforms on Azure Databricks and Azure cloud tech.
• Build robust batch/streaming ETL/ELT pipelines using PySpark/SQL and Azure-native tools.
• Architect Lakehouse environments with Delta Lake and medallion layers (Bronze/Silver/Gold).
• Collaborate with stakeholders to translate requirements into scalable data solutions.
• Oversee data ingestion, transformation, quality, governance, and storage optimization.

Skills

Python

PySpark

SQL

Scala

Delta Lake

Lakehouse architecture

Data governance

Azure Databricks

Azure cloud

PowerShell

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or related field

Tools

Azure Databricks

Kafka

Airflow

Azure DevOps

MLflow

Job description

The Cloud Platform Manager is a senior technical leadership role responsible for designing, building, optimizing, and managing scalable enterprise data platforms within a cloud-based Azure ecosystem. This position plays a critical role in driving enterprise data engineering, analytics, machine learning enablement, and business intelligence initiatives through the development of modern Lakehouse architectures and high-performance data pipelines.

Key responsibilities include leading data engineering teams, architecting scalable Azure Databricks solutions, and implementing secure, high-quality enterprise data platforms that support analytics, reporting, and operational decision‑making. The role combines hands‑on technical expertise with leadership, governance, and strategic planning responsibilities to support enterprise‑wide data initiatives.

Key Responsibilities

• Lead, mentor, and support teams of data engineers, analysts, and technical professionals while promoting development standards and best practices.
• Design, develop, deploy, and optimize scalable enterprise data platforms using Azure Databricks and Azure cloud technologies.
• Build and maintain robust batch and streaming ETL/ELT data pipelines using PySpark, Scala, SQL, and Azure-native tools.
• Architect modern Lakehouse environments utilizing Delta Lake and medallion architecture methodologies (Bronze, Silver, Gold layers).
• Collaborate with business stakeholders, data scientists, and technical teams to gather requirements and translate them into scalable technical solutions.
• Oversee data ingestion, transformation, integration, quality validation, governance, and storage optimization initiatives.
• Implement data governance frameworks, security controls, RBAC access models, lineage tracking, and compliance processes using Unity Catalog and related technologies.
• Integrate enterprise platforms with Azure Data Factory, Azure Data Lake Storage, Azure Synapse, Kafka, DevOps pipelines, and other cloud‑native services.
• Optimize Spark workloads, Databricks clusters, query performance, indexing strategies, partitioning, and cost efficiency within Azure cloud environments.
• Support machine learning operations and model deployment initiatives utilizing MLflow and related MLOps tools.
• Develop and maintain scalable data models, schemas, and enterprise analytics frameworks.
• Participate in troubleshooting, production support, incident management, and continuous improvement initiatives for enterprise data environments.
• Ensure data platform reliability, scalability, performance, and operational excellence across cloud infrastructure.

Minimum Education & Experience Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field required.
• Minimum of 5-7 years of hands‑on experience in data engineering, cloud architecture, or enterprise data platform development.
• Minimum of 2-4 years of direct experience working with Azure Databricks and Azure cloud data technologies.
• Prior experience leading or mentoring technical teams, including data engineers, analysts, or data scientists.
• Strong experience developing enterprise‑scale ETL/ELT pipelines and distributed data processing solutions.
• Experience with modern cloud‑based data platforms, big data technologies, and streaming architectures.
• Strong background in SQL, Python/PySpark, Scala, and enterprise data modeling principles.

Special Requirements

• Ability to work within a hybrid or remote enterprise technology environment.
• Availability to support production incidents or critical deployments outside standard business hours when necessary.
• Ability to work collaboratively across technical and non‑technical business teams.
• Participation in strategic planning, architecture reviews, and enterprise governance initiatives as required.
• Relevant cloud and data engineering certifications are preferred.

Knowledge, Skills, and Abilities

• Advanced knowledge of Delta Lake, Lakehouse architecture, and medallion data processing frameworks.
• Strong proficiency in Python, PySpark, SQL, Scala, and PowerShell.
• Expertise with Apache Spark runtime optimization and distributed data processing frameworks.
• Experience integrating real‑time and batch data processing systems using Kafka, Spark Streaming, and related technologies.
• Knowledge of relational and NoSQL database technologies and data architecture principles.
• Familiarity with file formats such as Parquet, ORC, and Avro, including storage optimization techniques.
• Understanding of CI/CD processes, DevOps methodologies, and orchestration tools such as Airflow and Azure DevOps.
• Strong understanding of data governance, security, lineage, RBAC, compliance, and enterprise data management best practices.
• Excellent analytical, troubleshooting, and problem‑solving abilities.
• Strong leadership, mentoring, organizational, and project coordination skills.
• Excellent written and verbal communication skills with the ability to communicate technical concepts to diverse audiences.
• Ability to manage multiple priorities in fast‑paced enterprise environments.

Additional Desired Characteristics

• Microsoft Azure Data Engineer, Azure Solutions Architect, or Databricks certifications preferred.
• Experience supporting enterprise analytics, reporting, and machine learning initiatives.
• Familiarity with MLflow, MLOps practices, and AI/analytics platforms.
• Experience with large‑scale enterprise data modernization initiatives.
• Exposure to legal, healthcare, financial services, or other highly regulated industries preferred.
• Strong understanding of enterprise cloud security, networking, and identity management concepts.
• Experience implementing enterprise‑wide data governance and data quality initiatives.