Senior Databricks Engineer Bengaluru Chubb Digital Technology

Chubb Deutschland GmbH; Bengaluru; On-site; INR 3,500,000 - 6,500,000; Full time

Job summary

Chubb Deutschland GmbH in Bengaluru seeks a senior Databricks Platform Engineer to design and operate scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog. You will build ELT/ETL pipelines, integrate with Azure services, and govern data for enterprise-scale analytics.

The role emphasizes CI/CD with Azure DevOps, infrastructure as code with Terraform/Bicep, and advanced data processing with PySpark and SQL. Collaboration and governance drive platform maturity.

Qualifications

• Databricks Certified Data Engineer Professional or Associate (mandatory).
• Microsoft Certified: Azure Data Engineer Associate (mandatory).
• 10+ years of hands-on experience in Data Engineering roles.
• Deep expertise in Databricks — Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
• Advanced PySpark and Python skills for large-scale distributed data processing.
• Strong SQL skills: query authoring, performance tuning, and data modeling.
• Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
• Hands-on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep).
• Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns.
• Experience with Structured Streaming or other real-time data processing frameworks.

Responsibilities

• Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
• Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
• Unity Catalog: Govern data assets — tables, volumes, and models — through Unity Catalog, enforcing fine-grained access control and lineage tracking.
• Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
• Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.
• ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
• Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
• Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.
• Code Quality: Write clean, efficient, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards.
• CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
• Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
• Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.
• Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations.
• Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
• Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.
• Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
• Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.
• Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.

Skills

Databricks

PySpark

Python

SQL

Azure

Tools

Azure DevOps

Terraform

Bicep

DABs

Delta Live Tables

Unity Catalog

Job description

Key Responsibilities

Databricks Platform Engineering

• Lakehouse Architecture: Design and implement scalable Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog as the primary platform.
• Delta Live Tables: Build and manage DLT pipelines for declarative, reliable, and maintainable data transformation workflows.
• Unity Catalog: Govern data assets — tables, volumes, and models — through Unity Catalog, enforcing fine-grained access control and lineage tracking.
• Performance Optimization: Tune Spark jobs and Delta tables using Z-ordering, liquid clustering, auto-optimize, and partition strategies to maximize query performance.
• Databricks Workflows: Orchestrate multi-task pipelines using Databricks Workflows, including job scheduling, dependency management, and alerting.

Data Pipeline & Integration

• ELT/ETL Pipelines: Design, develop, and maintain robust ELT/ETL pipelines ingesting data from APIs, databases, streaming sources, and flat files into the Lakehouse.
• Azure Integration: Integrate Databricks with Azure Data Factory, ADLS Gen2, Azure Synapse, Event Hubs, and other Azure-native services.
• Streaming: Build real-time data processing pipelines using Structured Streaming and Delta Live Tables for low-latency analytics use cases.

Software Engineering & DevOps

• Code Quality: Write clean, efficient, and reusable PySpark, Python, and SQL code following SOLID principles and team coding standards.
• CI/CD: Implement and manage CI/CD pipelines in Azure DevOps for automated testing, deployment, and release of Databricks notebooks and jobs.
• Infrastructure as Code: Provision and manage Databricks workspaces, clusters, and supporting Azure infrastructure using Terraform or Bicep.
• Databricks Asset Bundles: Package and deploy Databricks projects using DABs for reproducible, environment-consistent deployments.

Data Governance & Quality

• Data Quality Frameworks: Implement automated data validation, expectation checks, and quality monitoring using Great Expectations or DLT expectations.
• Data Lineage: Maintain end-to-end data lineage visibility through Unity Catalog and Microsoft Purview integration.
• Documentation: Produce and maintain thorough technical documentation for pipelines, data models, runbooks, and architectural decisions.

Collaboration & Leadership

• Mentorship: Coach and mentor junior and mid-level engineers, conducting code reviews and sharing best practices.
• Stakeholder Engagement: Translate business requirements into robust technical solutions; communicate trade-offs clearly to non-technical stakeholders.
• Standards & Governance: Contribute to and enforce team engineering standards, SDLC processes, and platform governance policies.

Required Qualifications

• Databricks Certified Data Engineer Professional or Associate (mandatory).
• Microsoft Certified: Azure Data Engineer Associate (mandatory).
• 10+ years of hands‑on experience in Data Engineering roles.
• Deep expertise in Databricks — Delta Lake, Unity Catalog, Delta Live Tables, Workflows, Clusters.
• Advanced PySpark and Python skills for large-scale distributed data processing.
• Strong SQL skills: query authoring, performance tuning, and data modeling.
• Experience with Azure Data Factory, ADLS Gen2, Azure Synapse Analytics.
• Hands‑on experience with Azure DevOps CI/CD and Infrastructure as Code (Terraform or Bicep).
• Strong understanding of Data Warehousing concepts, dimensional modeling, and Data Lakehouse patterns (Medallion architecture).
• Experience with Structured Streaming or other real‑time data processing frameworks.

Preferred Qualifications

• Experience with Microsoft Purview for data cataloguing and governance.
• Familiarity with dbt (data build tool) for SQL-based transformation workflows.
• Knowledge of MLflow and Databricks Feature Store for ML engineering integration.
• Exposure to Databricks Asset Bundles (DABs) for project packaging and deployment.
• Experience with Azure Function Apps, Logic Apps, or Event Hubs for event‑driven architectures.
• Background in data mesh or data product thinking.

Key Skills at a Glance

Core Platform Databricks, Delta Lake, Unity Catalog, DLT

Languages PySpark, Python, SQL

Cloud & Integration Azure Data Factory, ADLS Gen2, Synapse, Event Hubs

DevOps & IaC Azure DevOps, Terraform, Bicep, DABs

Data Quality & Governance Great Expectations, DLT Expectations, Purview

Architecture Lakehouse, Medallion Architecture, Data Mesh

Why Join Us?

• Lead the build‑out of a modern, Databricks‑first data platform at scale.
• Work with a collaborative, forward‑thinking Data & Analytics team.
• Access to cutting‑edge tooling across Azure, Databricks, and the broader modern data stack.
• Competitive salary, benefits, and a strong commitment to your professional development and certifications.
• Opportunity to shape team standards, architecture decisions, and engineering culture.
• Contribute to high‑impact, global data initiatives.

Each listing including the job title, location, brief description, and a link to a page with more details.