Azure Databricks Data Engineer

Birlasoft Limited; Pune District; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Birlasoft Limited is seeking an experienced Azure Databricks Developer to design and implement data solutions in the Azure data ecosystem. The role covers data solution design, MDW/Lakehouse architectures, and building scalable ETL/ELT pipelines with ADF, Databricks, PySpark, and SQL.

You will collaborate with BI and analytics teams to ensure data quality, security, and governance while optimizing performance and deploying artifacts through CI/CD pipelines in Azure DevOps.

Qualifications

• 7–10 years of data engineering experience in Azure data ecosystem.
• At least 2 years of real project experience in Azure Databricks.
• At least 2 years building data pipelines using Azure Data Factory.
• At least 2 years of PySpark-based transformations in Databricks.
• Strong SQL with performance tuning for large datasets.
• Experience with ADLS Gen2 and Delta/Parquet/JSON formats.
• Knowledge of CI/CD pipelines for automated deployment.

Responsibilities

• Data Solution Design & Architecture: design end to end data solutions using Azure Data Factory, Databricks, PySpark, SQL and other Azure services.
• Data Pipeline Development: build and optimize ELT/ETL pipelines for structured, semi-structured, and unstructured data.
• Databricks & PySpark Engineering: develop scalable PySpark transformations and manage Databricks clusters, jobs, notebooks, and CI/CD integrations.
• Data Governance, Quality & Security: implement data quality checks, audit mechanisms, and ensure data security compliance.
• Collaboration & Stakeholder Management: work with BI/analytics teams and cross-functional engineering teams to deliver reliable, production-ready solutions.

Skills

Azure Databricks

PySpark

SQL

Azure Data Factory

Delta Lake

Azure Data Lake Storage

CI/CD

Tools

Azure DevOps

Job description

Azure Databricks Developer

Area(s) of responsibility

Key Responsibilities

1. Data Solution Design & Architecture

   • Design end to end data engineering solutions using Azure Data Factory, Azure Databricks, PySpark, SQL, and other Azure-native services.
   • Architect and implement scalable and secure Modern Data Warehouse (MDW) and Lakehouse solutions leveraging Azure Data Lake Storage and Databricks.
   • Develop data models, integration patterns, and reusable frameworks aligned with best practices and enterprise architecture standards.
   • Participate in requirement discussions, solution blueprinting, and technical feasibility assessments.

2. Data Pipeline Development

   • Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
   • Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
   • Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
   • Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.

3. Databricks & PySpark Engineering

   • Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
   • Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
   • Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
   • Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

4. Data Governance, Quality & Security

   • Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency.
   • Ensure compliance with organizational standards for data security, encryption, access control, and data lifecycle management.
   • Create and maintain technical documentation, data flow diagrams, and operational support guides.

5. Collaboration & Stakeholder Management

   • Collaborate closely with BI, analytics, and business teams to understand data requirements and deliver reliable, production-ready solutions.
   • Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.
   • Provide guidance and mentoring to junior engineers when required.

Mandatory Skills & Experience

• Minimum 7–10 years of experience in data engineering, with strong hands on exposure to Azure data ecosystem.
• At least 2 years of real project experience in Azure Databricks (not POCs).
• At least 2 years of hands-on experience building data pipelines using Azure Data Factory (ADF).
• At least 2 years of experience developing PySpark-based transformations in Databricks.
• Strong SQL programming experience, including writing complex queries, performance tuning, and handling large datasets.
• Experience with Azure Data Lake Storage (ADLS Gen2), data partitioning strategies, and file formats like Parquet/Delta/JSON.
• Knowledge of CI/CD pipelines (Azure DevOps preferred) for automated deployment of ADF/Databricks artifacts.