Azure Databricks Developer

Birlasoft; Dadri; On-site; INR 2,600,000 - 4,000,000; Full time

Job summary

Birlasoft is seeking a Senior Azure Data Engineer/Technical Lead to design and deliver enterprise-scale data solutions on Azure. You will lead architecture, mentor teams, and collaborate with stakeholders to build scalable data ecosystems using Databricks, PySpark, ADF, and Delta Lake.

The role demands hands-on engineering, strong SQL, and extensive experience with MDW/Lakehouse architectures on Azure, ADLS, and CI/CD integration.

Qualifications

• 10–13 years of data engineering experience
• 3+ years in Azure Databricks
• 5+ years in Azure Data Factory (ADF)
• 2+ years PySpark experience
• 6+ years ETL/Data Warehouse experience
• Strong SQL expertise
• Experience with ADLS and Delta Lake

Responsibilities

• Lead design, development and deployment of large‑scale data solutions on Azure Databricks, PySpark, SQL, ADF and ADLS.
• Architect end-to-end Modern Data Warehouse and Lakehouse solutions with scalability, security and cost optimization.
• Define standards, reusable frameworks and architectural guidelines for engineering teams.
• Provide technical leadership across the project lifecycle.

Skills

Azure Databricks

PySpark

Azure Data Factory (ADF)

SQL

Data engineering

MDW/Lakehouse architecture

Delta Lake

ETL & Data Warehouse

Azure Data Lake Storage (ADLS)

Job description

Job Description:

Area(s) of responsibility

Role Summary
We are seeking a seasoned Senior Azure Data Engineer / Sr. Technical Lead with extensive expertise in Azure Databricks, PySpark, ADF, SQL, and modern data engineering practices. This role requires strong technical leadership, hands on engineering capabilities, and the ability to design, architect, and deliver enterprise-scale data solutions. The ideal candidate will lead complex data initiatives, mentor engineering teams, and collaborate with cross‑functional stakeholders to build a robust and scalable data ecosystem on Azure.
________________________________________
Key Responsibilities

1. Solution Architecture & Technical Leadership
   • Lead the design, development, and deployment of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, ADF, and Azure Data Lake.
   • Architect end‑to‑end Modern Data Warehouse (MDW) and Lakehouse solutions, ensuring scalability, performance, security, and cost optimization.
   • Define technical standards, coding best practices, reusable frameworks, and architectural guidelines for engineering teams.
   • Provide technical leadership across the project lifecycle—requirements analysis, solution blueprinting, estimation, development, and deployment.
2. Data Pipeline Engineering
   • Build, optimize, and maintain scalable, high performance ELT/ETL pipelines to process large volumes of structured and unstructured data.
   • Set up complex data ingestion frameworks, enabling seamless integration with on‑premise systems, cloud services, APIs, and third‑party sources.
   • Ensure high availability, data reliability, and error‑resilient orchestration workflows in Azure Data Factory.
3. Azure Databricks & PySpark Expertise
   • Design and implement advanced transformation logic using PySpark on Databricks, ensuring efficient data processing and code modularity.
   • Utilize Delta Lake capabilities—ACID transactions, schema evolution, versioning, time travel—to manage enterprise‑grade datasets.
   • Perform cluster‑level tuning, optimization of shuffle operations, caching, partitioning, and job parallelization.
   • Manage Databricks job pipelines, notebooks, clusters, job scheduling, and integration with CI/CD pipelines

Mandatory Skills & Experience

• 10–13 years of overall experience in data engineering and enterprise data platforms.
• Minimum 3 years of hands‑on project experience in Azure Databricks (beyond POCs).
• Minimum 5 years of experience building and orchestrating pipelines in Azure Data Factory (ADF).
• Minimum 2+ years of strong PySpark experience with complex data transformation logic.
• 6+ years of ETL & Data Warehouse experience, including dimensional modelling, data partitioning, and performance optimization.
• Strong SQL expertise—complex queries, optimization, stored procedures, analytical functions.
• Proven experience working with Azure Data Lake Storage (ADLS), Delta Lake, and modern data processing patterns.

Good‑to‑Have Skills

• Experience working in Agile/Scrum environments, including sprint planning and backlog management.
• Knowledge of Azure Synapse, Event Hub, Databricks workflows, and Azure DevOps CI/CD.
• Databricks certifications (Associate/Professional) or Azure certifications (DP 203/DP 900).