Lead solution architect - Data engineering

bp; Pune District; Hybrid; INR 2,500,000 - 4,500,000; Full time

Benefits offered by this job

Health insurance

Medical care package

Relocation assistance within India

Job summary

bp is seeking an experienced Data Architect to design and optimize cloud‑native data platforms on Azure. The role requires leadership in governance, data modeling, and scalable ETL/ELT pipelines across a large data estate.

You will mentor teams, drive data strategy with internal stakeholders, and ensure GDPR compliance and secure data sharing. Hybrid work in India is supported with relocation options within the country.

Qualifications

• 7+ years in Data Architecture or Data Engineering roles.
• Expert‑level knowledge of Azure Databricks (Spark, Delta Lake, MLflow).
• Azure Data Lake Gen2, Azure Data Factory / Synapse Pipelines.
• Strong programming in PySpark, SQL, and Python.
• Hands‑on with CI/CD (Azure DevOps / GitHub Actions).
• Deep understanding of distributed systems & big data performance tuning.

Responsibilities

• Design end‑to‑end cloud‑native data platforms using Azure services.
• Lead Delta Lake governance, data modeling, and metadata management.
• Architect Delta Lake medallion architectures and scalable ETL/ELT pipelines.
• Collaborate with business, product owners, and engineers to drive data strategy.

Skills

Azure Databricks

Azure Data Lake Gen2

Azure Data Factory / Synapse Pipelines

Azure SQL / Synapse / Cosmos DB

Unity Catalog & Delta Live Tables

PySpark/SQL/Python

CI/CD (Azure DevOps / GitHub Actions)

Distributed systems performance tuning

Education

Bachelor's degree in Computer Science or related field

Tools

Azure Purview

Job description

Entity:

Technology

Job Family Group:

IT&S Group

Job Description:

Role Overview

We are a global energy business involved in every aspect of the energy system. We are working towards delivering light, heat and mobility to millions of people, every day. We are one of the very few companies equipped to solve some of the big complex challenges that matter for the future. We have a real contribution to make to the world’s ambition of a low carbon future. Join us and be part of what we can accomplish together. You can participate in our new ambition to become a net zero company by 2050 or sooner, and to help the world get to net zero.

Critical to achieving bp’s digital ambitions is the delivery of our high value data and analytics initiatives, and the enablement of the technologies and platforms that will support those objectives. We are developing deep data capability to transform the access, supply, control and quality to our vast and ever-growing reserves of data.

We are looking for an outstanding Data Architect with deep expertise in Microsoft Azure and Databricks to design, build, and optimize modern data platforms. The ideal candidate will have strong architectural leadership, hands‑on technical proficiency, and the ability to influence internal customers to drive data strategy, governance, and scalable data solutions.. These use cases would drive significant operational efficiency, safety and environmental aims that are key to the transformation of bp.

Key Responsibilities

Data Architecture & Platform Design

• Architect end-to-end cloud-native data platforms using Azure services, including Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse, Azure SQL, Azure Data Factory, and Event/Service Bus.


• Define and implement data modeling standards, including dimensional, canonical, and domain-driven designs.


• Design Delta Lake architectures, including medallion (Bronze–Silver–Gold) frameworks.


• Develop scalable and secure ETL/ELT pipelines using Databricks (PySpark/Spark SQL).

2. Databricks & Delta Lake Engineering

• Implement Delta Lake best practices, including OPTIMIZE, ZORDER, Change Data Feed (CDF), schema evolution, and table versioning.


• Optimize Spark workloads for performance, cost, and cluster utilization.


• Lead governance implementation using Unity Catalog, including permissions, data lineage, and metadata modeling.

3. Azure Data Ecosystem Integration

• Integrate solutions across Azure Data Factory, Synapse Pipelines, Event Hubs, Functions, Key Vault, Azure DevOps, and Purview.


• Drive CI/CD and Infrastructure-as-Code using Terraform, Bicep, or ARM templates.


• Build APIs and microservices supporting data consumption.

4. Data Governance & Security

• Define policies for data quality, master data management, metadata, and data lifecycle management.


• Ensure GDPR-compliant data design, encryption (KMS/Key Vault), RBAC/ABAC, and secure network architecture.


• Establish frameworks for observability, including logging, monitoring, and alerting.

5. Stakeholder Leadership

• Collaborate with business, product owners, engineers, and analysts to gather data requirements.


• Lead architectural reviews and provide long‑term technology strategy.


• Mentor engineering teams on Azure, Databricks, and scalable data design principles.

Required Skills & Qualifications

Technical Skills

• 7+ years in Data Architecture or Data Engineering roles.


• Expert‑level knowledge of:


• Azure Databricks (Spark, Delta Lake, MLflow), Agent Bricks


• Azure Data Lake Gen2


• Azure Data Factory / Synapse Pipelines


• Azure SQL / Synapse Dedicated SQL / Cosmos DB


• Unity Catalog & Delta Live Tables


• Strong programming experience in PySpark, SQL, and Python.


• Hands‑on withCI/CD (Azure DevOps / GitHub Actions).


• Deep understanding of distributed systems & big data performance tuning.

Soft Skills

• Ability to translate business requirements into technical architecture.


• Strong documentation, presentation, and interpersonal skills.


• Leadership in multi‑disciplinary teams and sophisticated enterprise environments.

Formal education

• A Bachelor’s (or higher) degree, ideally in Computer Science, MIS/IT, Mathematics, chartered accountancy, engineering, or equivalent work experience.

Preferred Qualifications and Skills

• Azure certifications (e.g., DP-203, AZ-305, Databricks Data Engineer Professional/Associate).


• Courses on Generative AI in Data engineering


• Experience with data mesh, domain-driven design, and event-driven architecture.


• Familiarity with Power BI or other BI tools.


• Experience integrating governance tools such as Azure Purview.


• TOGAF 9 certification

Why join us?

At bp, we provide an excellent working environment and employee benefits! We offer an open and inclusive culture and a great work‑life balance. Tremendous learning and development opportunities are available to craft a career path, life and health insurance, medical care package and many others!

We support our people to learn and grow in a diverse and exciting environment. We believe that our team is strengthened by diversity. We are committed to crafting an inclusive environment in which everyone is respected and treated fairly. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, perform crucial job functions, and receive other benefits and privileges of employment.

Travel Requirement

Up to 10% travel should be expected with this role

Relocation Assistance:

This role is eligible for relocation within country

Remote Type:

This position is a hybrid of office/remote working

Skills:

Agility core practices, Business modelling, Creativity and Innovation, Data Analysis, Data design, Digital Analytics, Emerging technology monitoring, Information Assurance, Integration, IT Management, IT strategy and planning, Network Design, Requirements definition and management, Risk Management, Service operations and resiliency, Solution Architecture, Systems Development Management, Technical specialism, User Experience Analysis

Legal Disclaimer:

We are an equal opportunity employer. We do not discriminate on the basis of protected characteristics like race, religion, color, sex, national origin, sexual orientation, veteran status or disability status.Individuals with an accessibility need may request an adjustment/accommodation related to bp’s recruiting process (e.g., accessing the job application, completing required assessments, participating in telephone screenings or interviews, etc.). If you would like to request an adjustment/accommodation related to the recruitment process,please contact us.

If you are selected for a position and depending upon your role, your employment may be contingent upon adherence to local policy. This may include pre-placement drug screening, medical review of physical fitness for the role, and background checks.