Azure Databricks Specialist

Birlasoft; Pune District; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

Birlasoft is seeking a Databricks Data Engineer for our Pune office to architect and optimize data platforms on Azure. You will build high-throughput ELT/ETL pipelines using Databricks, PySpark, ADF, and SQL.

Responsibilities include ingestion, Bronze-to-Silver transformations, data quality checks, and integration from on-prem and cloud sources. Collaborate with IT and business teams to deliver production-ready solutions.

Qualifications

• Minimum 4+ years of data engineering in Azure data ecosystem.
• At least 3 years of real project experience in Databricks on Azure (ADLS, Delta Lake, Spark, Unity Catalog).
• At least 2 years building data pipelines using Azure Data Factory (ADF).
• At least 2 years developing PySpark-based transformations in Databricks.
• Strong SQL with complex queries and performance tuning on large data sets.

Responsibilities

• Build and maintain ingestion frameworks (ADF / Databricks / Spark).
• Implement Bronze -> Silver transformations aligned to architecture.
• Architect data quality checks, schema validation, and contract rules.
• Build and optimize robust, high-throughput ELT/ETL pipelines.
• Integrate data from on-premise and cloud systems, APIs and third-party sources.
• Implement PySpark transformations for performance and modularity.
• Build orchestration workflows in ADF with pipelines and triggers.
• Familiar with using Databricks Genie for deployment and management.

Skills

Databricks

PySpark

ADF

SQL

Delta Lake

Spark

Databricks Genie

Tools

Azure Data Factory (ADF)

Databricks

Delta Lake

Job description

Select how often (in days) to receive an alert:

Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI

Description:

Area(s) of responsibility

Skills: Azure Databricks Engineer

Experience: 4-7 Years
Location: Pune, Mumbai, Noida, Chennai, Bangalore

Role Summary

We are looking for a highly skilled Databricks Data Engineer with strong hands‑on expertise in Databricks, PySpark, ADF, and SQL. The candidate will be responsible for architecting, developing, and optimizing modern data platforms and analytical solutions on Azure. The role requires strong experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Key Responsibilities

• Data Engineering Execution
• Build and maintain ingestion frameworks (ADF / Databricks / Spark)
  Implement Bronze -> Silver transformations aligned to architecture
  Architect data quality checks, schema validation, and contract rules
• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi-structured, and unstructured data.
• Integrate data from multiple on‑premise and cloud‑based systems, APIs, and third‑party sources.
• Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
• Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
• Familiar with using Databicks Genie.

Databricks & PySpark Engineering

• Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
• Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
• Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
• Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

Collaboration & Stakeholder Management

• Collaborate closely with customer IT and business teams to understand data requirements and deliver reliable, production-ready solutions.
• Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.
• Provide guidance and mentoring to junior engineers when required.

Mandatory Skills & Experience

• Minimum 4+ years of experience in data engineering, with strong hands‑on exposure to Azure data ecosystem.
• At least 3 years of real project experience in Databricks (Azure Data Platform (ADF, ADLS, Azure SQL) & Databricks (DLT, Delta Lake, Spark, Workflows, Unity Catalog).
• At least 2 years of hands‑on experience building data pipelines using Azure Data Factory (ADF).
• At least 2 years of experience developing PySpark-based transformations in Databricks.
• Strong SQL programming experience, including writing complex queries, performance tuning, and handling large datasets.

Knowledge of CI/CD pipelines (Azure DevOps preferred) for automated deployment of ADF/Databricks artifacts