Senior Databricks Data Engineer

Unison Group; Bengaluru; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Unison Group in Bengaluru, India, is seeking a data engineer to build and maintain scalable data pipelines using Azure Databricks for DATA & AI ACDP projects. You will apply 3-4 years of experience to deliver reliable ETL processes and collaborate on data-driven insights.

Responsibilities include implementing Medallion Architecture with Delta Lake, using PySpark/SQL/Python, and enabling CI/CD for Databricks deployments with Azure DevOps and ADF integration.

Qualifications

• > 8 years in data engineering with focus on Azure Databricks pipelines
• "> Proficiency in PySpark, SQL, Python
• "> Experience with cloud storage and basic orchestration
• "> Familiarity with Azure fundamentals and collaboration tools

Responsibilities

• Design, build, and optimize data pipelines in Azure Databricks for ingestion, ETL/ELT, and transformations with Delta Lake
• Implement Medallion Architecture (Bronze, Silver, Gold) for data quality and versioning
• Utilize Databricks Spark (PySpark, SQL), Delta Live Tables, and Unity Catalog for governance
• Integrate data from ADLS Gen2, Azure SQL Database, and SQL Pools; support Databricks workflows
• Implement CI/CD in Azure DevOps for Databricks deployments, including Bicep templates and ADF integration
• Apply data quality measures, security (RBAC, encryption), and monitoring for performance/cost efficiency
• Collaborate with analysts to refine requirements into workflows; handle data cleansing, modeling, and Python/SQL scripting
• Troubleshoot pipelines and contribute to documentation/best practices

Skills

PySpark

SQL

Python

Azure Databricks

Delta Lake

Delta Live Tables

Unity Catalog

RBAC

CI/CD

Tools

Azure Databricks

Azure DevOps

Bicep templates

ADF

Job description

Position Overview:

Build and maintain scalable data pipelines using Azure Databricks to support DATA & AI ACDP projects. Apply 3-4 years of experience to deliver reliable ETL processes and collaborate on data-driven insights.

Interview - Face to face

Position Overview:

Build and maintain scalable data pipelines using Azure Databricks to support DATA & AI ACDP projects. Apply 3-4 years of experience to deliver reliable ETL processes and collaborate on data-driven insights.

Key Responsibilities:

• Design, build, and optimize data pipelines in Azure Databricks for ingestion, ETL/ELT, and transformations, implementing Medallion Architecture (Bronze, Silver, Gold) with Delta Lake for data quality and versioning
• Utilize Databricks Spark (PySpark, SQL), Delta Live Tables, and Unity Catalog for pipeline development, governance, and basic streaming
• Integrate data from ADLS Gen2, Azure SQL Database, and SQL Pools; support orchestration via Databricks workflows
• Implement CI/CD in Azure DevOps for Databricks deployments, including Bicep templates and ADF integration
• Apply data quality measures (expectations, constraints), security (RBAC, encryption), and monitoring for performance/cost efficiency using auto-scaling and Photon
• Work with analysts and stakeholders to refine requirements into functional workflows; handle data cleansing, modeling, and Python/SQL scripting for structured/unstructured data
• Troubleshoot pipelines and contribute to documentation/best practices

Required Qualifications:

• 8 years in data engineering, focused on Azure Databricks pipelines and ETL
• Proficiency in PySpark, SQL, Python; experience with cloud storage and basic orchestration
• Familiarity with Azure fundamentals and collaborative tools