Sr Data Engineer Databricks

Unison Group; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Unison Group in Bengaluru is seeking a senior data engineer leader to architect and optimize enterprise-scale data pipelines on Azure Databricks, adopting Delta Lake and Medallion Architecture. You will mentor teams, drive CI/CD with Azure DevOps and Bicep, ensure data quality, governance, and security, and collaborate with data scientists and executives to deliver scalable solutions.

This role requires 7-8+ years' experience, strong PySpark/Scala, cloud-native design, and a passion for

Qualifications

• 7-8+ years in data engineering with Azure Databricks.
• Strong Python/PySpark, SQL, Scala and data architecture skills.
• Experience in cloud-native solutions, CI/CD, and mentoring.

Responsibilities

• Architect end-to-end data pipelines using Azure Databricks with Delta Lake.
• Lead and mentor teams, ensure scalability, and drive AI/ML integration.
• Integrate with ADLS Gen2, Azure SQL Database, and Azure SQL pools.
• Design CI/CD pipelines with Azure DevOps and IaC using Bicep.
• Ensure data quality, governance, RBAC, and security best practices.
• Monitor and optimize petabyte-scale streaming and batch workloads.
• Collaborate with analysts and executives to translate requirements into reusable components.

Skills

Azure Databricks

Python/PySpark

SQL

Scala

Data Architecture

Distributed Systems

CI/CD

Mentoring

Communication

Tools

Delta Lake

Unity Catalog

Azure DevOps

Bicep

Databricks

Job description

Job Description:

Position Overview:

• Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for ACDP projects, driving innovation in DATA & AI initiatives.
• Leverage 7-8 years of expertise to mentor teams, ensure scalability, and deliver mission-critical data solutions.

Key Responsibilities:

• Architect and implement end-to-end data pipelines with Azure Databricks (notebooks, jobs, clusters) for ingestion, ETL/ELT, and complex transformations, incorporating Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for ACID compliance, versioning, schema evolution, and data quality.
• Provide hands-on leadership in Databricks Spark (PySpark, Scala, SQL), Delta Live Tables, Structured Streaming, and Unity Catalog for advanced governance, security, and multi-workspace management.
• Integrate with Azure ecosystem including ADLS Gen2, Azure SQL Database, Serverless/Dedicated SQL Pools; orchestrate via Databricks workflows and hybrid ADF triggers for high-volume ACDP workloads.
• Design and deploy CI/CD pipelines using Azure DevOps, Bicep for infrastructure-as code, and advanced automation for Databricks workspaces, emphasizing zero downtime deployments and cost optimization.
• Champion data quality, validation, cleansing with Databricks expectations/constraints, and compliance via RBAC, encryption, and Purview; mentor on best practices for Photon engine, auto-scaling, spot instances, and performance tuning
• Collaborate with senior analysts, data scientists, and executives to translate complex business requirements into reusable Databricks components, fostering AI/ML integration.
• Drive monitoring, troubleshooting, and optimization for petabyte-scale processing, including unstructured data handling, data modeling, and streaming from Kafka/Event Hubs

Required Qualifications:

• 7-8+ years in data engineering, with deep Azure Databricks expertise for big data processing in high-demand environments
• Advanced proficiency in Python/PySpark, SQL, Scala; strong data architecture and distributed systems experience
• Proven track record in cloud-native solutions (AZ-900+ preferred), CI/CD, and mentoring junior engineers
• Excellent communication for stakeholder alignment and documentation

Preferred: Power BI, Power Automate, Azure ML, enterprise governance tools.

Requirements: