Data Engineer (Databricks ) Senior Manager

PwC

Pune District, Gurugram District, Bengaluru

Hybrid

INR 3,500,000 - 5,500,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

PwC in Pune seeks a senior Data Platform Architect to own the enterprise Databricks strategy, establish a Center of Excellence, and lead governance and Lakehouse initiatives.

You will drive cloud-native architectures on AWS/Azure, design scalable data pipelines, optimize costs, ensure security and compliance, and mentor a global team of engineers. Collaborating with stakeholders, you will translate business needs into robust data strategies.

Responsibilities

  • Define and own the enterprise-wide Databricks platform strategy, roadmap, and adoption model in line with business goals.
  • Set up the Databricks Center of Excellence (CoE), including best practices, reusable frameworks, and engineering standards.
  • Assess new Databricks capabilities (Unity Catalog, Delta Live Tables, Serverless, Photon, Mosaic AI, Databricks SQL) and lead their adoption.
  • Drive platform cost optimization (FinOps), workspace strategy, and capacity planning.
  • Act as the primary technical liaison with Databricks and cloud vendor partners.
  • Design and implement an enterprise-grade CI/CD framework using Databricks Asset Bundles for consistent, repeatable deployments.
  • Standardize Infrastructure-as-Code and Configuration-as-Code practices across environments (Dev, QA, UAT, Prod).
  • Integrate DABs with DevOps tools such as Azure DevOps, GitHub Actions, GitLab CI, Jenkins, and Terraform.
  • Build reusable bundle templates, deployment patterns, and automated testing and validation pipelines.
  • Ensure version control, environment promotion, and release governance across all Databricks workloads.
  • Lead the design and rollout of data governance using Unity Catalog, covering access controls, data lineage, auditing, and data discovery.
  • Implement data security, privacy, and compliance controls (e.g., GDPR, HIPAA, SOC 2, CCPA, as applicable).
  • Define data quality frameworks, metadata management standards, and data ownership and stewardship models.
  • Work with InfoSec, Risk, and Compliance teams to enforce policies such as row/column-level security, data masking, and fine-grained permissions.
  • Set up monitoring, observability, and audit mechanisms across the data platform.
  • Architect and evolve the enterprise Data Lakehouse using Delta Lake and the Medallion Architecture (Bronze, Silver, Gold).
  • Design scalable data models that support analytics, BI, ML/AI, and real-time use cases.
  • Lead migrations from legacy data warehouses, Hadoop, and on-prem platforms to the Databricks Lakehouse.
  • Ensure performance tuning, storage optimization (Z-Ordering, Liquid Clustering, partitioning), and high availability.
  • Enable data sharing through Delta Sharing and support data mesh and data product approaches where relevant.
  • Design secure, scalable, and resilient cloud-native data architectures on AWS and/or Azure.
  • Oversee the integration of Databricks with cloud services, such as: Azure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Synapse, Entra ID, Purview; AWS: S3, Glue, Kinesis, Lambda, IAM, Redshift, Lake Formation
  • Define networking, identity, and security architecture, including VNet/VPC injection, Private Link, and SSO/SCIM.
  • Drive disaster recovery, business continuity, and multi-region strategies.
  • Lead the design and delivery of robust batch and streaming pipelines using PySpark, Spark SQL, Structured Streaming, Delta Live Tables, and Auto Loader.
  • Establish reusable ingestion, transformation, and orchestration frameworks (Databricks Workflows, Airflow, ADF).
  • Ensure pipelines meet SLAs, scalability, reliability, and data quality requirements.
  • Promote engineering excellence through code reviews, testing standards, modular design, and documentation.
  • Enable real-time and near-real-time data processing for critical business use cases.
  • Manage a portfolio of data engineering programs and projects, ensuring delivery on scope, schedule, budget, and quality.
  • Prioritize initiatives based on business value, ROI, and strategic alignment.
  • Own resource planning, budgeting, vendor management, and risk mitigation across the portfolio.
  • Report progress, KPIs, and outcomes to senior leadership and executive stakeholders.
  • Run delivery governance using Agile, SAFe, or hybrid methodologies.
  • Lead, mentor, and grow a team of data engineers, architects, and tech leads, including onshore and offshore teams.
  • Build a culture of innovation, accountability, and continuous learning.
  • Drive hiring, performance management, and career development for the team.

Job description

Key Responsibilities

1. Databricks Enterprise Strategy

  • Define and own the enterprise-wide Databricks platform strategy, roadmap, and adoption model in line with business goals.
  • Set up a Databricks Center of Excellence (CoE), including best practices, reusable frameworks, and engineering standards.
  • Assess new Databricks capabilities (Unity Catalog, Delta Live Tables, Serverless, Photon, Mosaic AI, Databricks SQL) and lead their adoption.
  • Drive platform cost optimization (FinOps), workspace strategy, and capacity planning.
  • Act as the primary technical liaison with Databricks and cloud vendor partners.

2. Advanced DABs (Databricks Asset Bundles) Framework

  • Design and implement an enterprise-grade CI/CD framework using Databricks Asset Bundles for consistent, repeatable deployments.
  • Standardize Infrastructure-as-Code and Configuration-as-Code practices across environments (Dev, QA, UAT, Prod).
  • Integrate DABs with DevOps tools such as Azure DevOps, GitHub Actions, GitLab CI, Jenkins, and Terraform.
  • Build reusable bundle templates, deployment patterns, and automated testing and validation pipelines.
  • Ensure version control, environment promotion, and release governance across all Databricks workloads.

3. Governance Implementation

  • Lead the design and rollout of data governance using Unity Catalog, covering access controls, data lineage, auditing, and data discovery.
  • Implement data security, privacy, and compliance controls (e.g., GDPR, HIPAA, SOC 2, CCPA, as applicable).
  • Define data quality frameworks, metadata management standards, and data ownership and stewardship models.
  • Work with InfoSec, Risk, and Compliance teams to enforce policies such as row/column-level security, data masking, and fine-grained permissions.
  • Set up monitoring, observability, and audit mechanisms across the data platform.

4. Data Lakehouse Architecture

  • Architect and evolve the enterprise Data Lakehouse using Delta Lake and the Medallion Architecture (Bronze, Silver, Gold).
  • Design scalable data models that support analytics, BI, ML/AI, and real-time use cases.
  • Lead migrations from legacy data warehouses, Hadoop, and on-prem platforms to the Databricks Lakehouse.
  • Ensure performance tuning, storage optimization (Z-Ordering, Liquid Clustering, partitioning), and high availability.
  • Enable data sharing through Delta Sharing and support data mesh and data product approaches where relevant.

5. Cloud Architecture (AWS / Azure)

  • Design secure, scalable, and resilient cloud-native data architectures on AWS and/or Azure.
  • Oversee the integration of Databricks with cloud services, such as:
    • Azure: ADLS Gen2, Azure Data Factory, Event Hubs, Key Vault, Synapse, Entra ID, Purview
    • AWS: S3, Glue, Kinesis, Lambda, IAM, Redshift, Lake Formation
  • Define networking, identity, and security architecture, including VNet/VPC injection, Private Link, and SSO/SCIM.
  • Drive disaster recovery, business continuity, and multi-region strategies.

6. Enterprise Data Pipeline Design

  • Lead the design and delivery of robust batch and streaming pipelines using PySpark, Spark SQL, Structured Streaming, Delta Live Tables, and Auto Loader.
  • Establish reusable ingestion, transformation, and orchestration frameworks (Databricks Workflows, Airflow, ADF).
  • Ensure pipelines meet SLAs, scalability, reliability, and data quality requirements.
  • Promote engineering excellence through code reviews, testing standards, modular design, and documentation.
  • Enable real-time and near-real-time data processing for critical business use cases.

7. Portfolio Management

  • Manage a portfolio of data engineering programs and projects, ensuring delivery on scope, schedule, budget, and quality.
  • Prioritize initiatives based on business value, ROI, and strategic alignment.
  • Own resource planning, budgeting, vendor management, and risk mitigation across the portfolio.
  • Report progress, KPIs, and outcomes to senior leadership and executive stakeholders.
  • Run delivery governance using Agile, SAFe, or hybrid methodologies.

People Leadership

  • Lead, mentor, and grow a team of data engineers, architects, and tech leads, including onshore and offshore teams.
  • Build a culture of innovation, accountability, and continuous learning.
  • Drive hiring, performance management, and career development for the team.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Technical Lead
Technical Lead

Carelon Global Solutions • Gurugram District, Bengaluru

Hybrid
INR 3,000,000 - 5,500,000
Data Architect I
Data Architect I

UST • Pune District

On-site
INR 4,000,000 - 7,000,000
Databricks Senior Architect
Databricks Senior Architect

Vrinda Global • Bengaluru

On-site
INR 3,000,000 - 6,000,000
Databricks Engineer
Databricks Engineer

Fa Ewjt Saasfaprod1 • Chennai

On-site
INR 3,500,000 - 6,000,000
Lead Data Engineer
Lead Data Engineer

Saur Energy International • Bengaluru

On-site
INR 4,000,000 - 7,000,000
Senior Architect_Databricks
Senior Architect_Databricks

Tredence • Kolkata District, Pune District, Chennai

Hybrid
INR 3,500,000 - 7,500,000
Resident Solution Architect
Resident Solution Architect

Celebal Technologies • Dadri

On-site
INR 3,000,000 - 6,000,000
Senior Databricks Engineer
Senior Databricks Engineer

Value Vision • Bengaluru

On-site
INR 4,200,000 - 6,600,000
Data Engineer, Data Platforms
Data Engineer, Data Platforms

S&P Global Market Intelligence • Hyderabad, Chennai

On-site
INR 2,500,000 - 4,500,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Bestal • Hyderabad

Hybrid
INR 4,000,000 - 7,000,000