Cloud Platform Manager

Confidentiel; Pittsburgh (Allegheny County); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Confidentiel in the United States seeks a Cloud Platform Manager to design, build, and manage scalable Azure-based data platforms. You will lead data engineers and analysts, architect Lakehouse environments with Delta Lake, and enable analytics, BI, and ML initiatives using PySpark, SQL, and Spark optimization.

The role blends hands-on coding with strategic governance, RBAC, and data quality controls, collaborating with stakeholders across the business.

Qualifications

• 5–7 years hands-on data engineering, cloud architecture, or enterprise data platform development.
• 2–4 years with Azure Databricks and Azure cloud data tech.

Responsibilities

• Lead data engineering teams and mentor analysts and engineers.
• Architect scalable Azure Databricks solutions and modern Lakehouse environments.
• Build and optimize batch and streaming ETL/ELT pipelines using PySpark, SQL, and Python.
• Implement data governance, RBAC, lineage, and security controls in Unity Catalog.
• Collaborate with stakeholders to translate requirements into scalable data solutions.
• Ensure reliability, performance, and cost efficiency of cloud data platforms.
• Support ML initiatives and model deployment with MLOps practices.

Skills

Leadership

Mentoring

SQL

Python

PySpark

Scala

PowerShell

Data modeling

Communication

Problem solving

Education

Bachelor's degree in a technical field

Tools

Azure Databricks

Azure Data Factory

Azure Data Lake Storage

Azure Synapse

Delta Lake

MLflow

Airflow

Azure DevOps

Kafka

Job description

The Cloud Platform Manager is a senior technical leadership role responsible for designing, building, optimizing, and managing scalable enterprise data platforms within a cloud-based Azure ecosystem. This position plays a critical role in driving enterprise data engineering, analytics, machine learning enablement, and business intelligence initiatives through the development of modern Lakehouse architectures and high-performance data pipelines.

Key responsibilities include leading data engineering teams, architecting scalable Azure Databricks solutions, and implementing secure, high-quality enterprise data platforms that support analytics, reporting, and operational decision‑making. The role combines hands‑on technical expertise with leadership, governance, and strategic planning responsibilities to support enterprise‑wide data initiatives.

Key Responsibilities

• Lead, mentor, and support teams of data engineers, analysts, and technical professionals while promoting development standards and best practices.
• Design, develop, deploy, and optimize scalable enterprise data platforms using Azure Databricks and Azure cloud technologies.
• Build and maintain robust batch and streaming ETL/ELT data pipelines using PySpark, Scala, SQL, and Azure-native tools.
• Architect modern Lakehouse environments utilizing Delta Lake and medallion architecture methodologies (Bronze, Silver, Gold layers).
• Collaborate with business stakeholders, data scientists, and technical teams to gather requirements and translate them into scalable technical solutions.
• Oversee data ingestion, transformation, integration, quality validation, governance, and storage optimization initiatives.
• Implement data governance frameworks, security controls, RBAC access models, lineage tracking, and compliance processes using Unity Catalog and related technologies.
• Integrate enterprise platforms with Azure Data Factory, Azure Data Lake Storage, Azure Synapse, Kafka, DevOps pipelines, and other cloud‑native services.
• Optimize Spark workloads, Databricks clusters, query performance, indexing strategies, partitioning, and cost efficiency within Azure cloud environments.
• Support machine learning operations and model deployment initiatives utilizing MLflow and related MLOps tools.
• Develop and maintain scalable data models, schemas, and enterprise analytics frameworks.
• Participate in troubleshooting, production support, incident management, and continuous improvement initiatives for enterprise data environments.
• Ensure data platform reliability, scalability, performance, and operational excellence across cloud infrastructure.

Minimum Education & Experience Requirements

• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field required.
• Minimum of 5-7 years of hands‑on experience in data engineering, cloud architecture, or enterprise data platform development.
• Minimum of 2-4 years of direct experience working with Azure Databricks and Azure cloud data technologies.
• Prior experience leading or mentoring technical teams, including data engineers, analysts, or data scientists.
• Strong experience developing enterprise‑scale ETL/ELT pipelines and distributed data processing solutions.
• Experience with modern cloud‑based data platforms, big data technologies, and streaming architectures.
• Strong background in SQL, Python/PySpark, Scala, and enterprise data modeling principles.

Special Requirements

• Ability to work within a hybrid or remote enterprise technology environment.
• Availability to support production incidents or critical deployments outside standard business hours when necessary.
• Ability to work collaboratively across technical and non‑technical business teams.
• Participation in strategic planning, architecture reviews, and enterprise governance initiatives as required.
• Relevant cloud and data engineering certifications are preferred.

Knowledge, Skills, and Abilities

• Advanced knowledge of Delta Lake, Lakehouse architecture, and medallion data processing frameworks.
• Strong proficiency in Python, PySpark, SQL, Scala, and PowerShell.
• Expertise with Apache Spark runtime optimization and distributed data processing frameworks.
• Experience integrating real‑time and batch data processing systems using Kafka, Spark Streaming, and related technologies.
• Knowledge of relational and NoSQL database technologies and data architecture principles.
• Familiarity with file formats such as Parquet, ORC, and Avro, including storage optimization techniques.
• Understanding of CI/CD processes, DevOps methodologies, and orchestration tools such as Airflow and Azure DevOps.
• Strong understanding of data governance, security, lineage, RBAC, compliance, and enterprise data management best practices.
• Excellent analytical, troubleshooting, and problem‑solving abilities.
• Strong leadership, mentoring, organizational, and project coordination skills.
• Excellent written and verbal communication skills with the ability to communicate technical concepts to diverse audiences.
• Ability to manage multiple priorities in fast‑paced enterprise environments.

Additional Desired Characteristics

• Microsoft Azure Data Engineer, Azure Solutions Architect, or Databricks certifications preferred.
• Experience supporting enterprise analytics, reporting, and machine learning initiatives.
• Familiarity with MLflow, MLOps practices, and AI/analytics platforms.
• Experience with large‑scale enterprise data modernization initiatives.
• Exposure to legal, healthcare, financial services, or other highly regulated industries preferred.
• Strong understanding of enterprise cloud security, networking, and identity management concepts.
• Experience implementing enterprise‑wide data governance and data quality initiatives.