Databricks & PySpark - Sr Technical Lead-Data Engg - Birlasoft

OpenTalent; Maharashtra; On-site; INR 1,800,000 - 2,600,000; Full time

Job summary

OpenTalent seeks a Sr Technical Lead-Data Engg to architect and deliver scalable data solutions on Azure using Databricks and PySpark. The role focuses on building enterprise-grade data pipelines, ingesting diverse sources, and implementing complex transformations with a strong emphasis on data quality and governance.

The candidate will lead design and optimization of ELT/ETL processes, manage Databricks clusters and CI/CD integrations, and work closely with platform teams to ensure reusable,

Qualifications

• Hands-on expertise with Databricks, PySpark, ADF and SQL.

Responsibilities

• Architect, develop and optimize modern data platforms on Azure.
• Build robust data pipelines ingesting from diverse sources and apply ETL/ELT transformations.
• Ensure data quality, governance and scalable analytics across domains.

Skills

Databricks

PySpark

ADF

SQL

Tools

Databricks

Spark

Delta Lake

Job description

Country/Region: IN

Requisition ID: 38968

Work Model:

Position Type:

Salary Range:

Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI

Title: Databricks & PySpark - Sr Technical Lead-Data Engg

Area(s) of responsibility

Role Summary

We are looking for a highly skilled Databricks Data Engineer with strong hands on expertise in Databricks, PySpark, ADF, and SQL. The candidate will be responsible for architecting, developing, and optimizing modern data platforms and analytical solutions on Azure. The role requires strong experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Key Responsibilities

1. Data Engineering Execution

• Build and maintain ingestion frameworks (ADF / Databricks / Spark)
• Implement Bronze -> Silver transformations aligned to architecture
• Architect data quality checks, schema validation, and contract rules
• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
• Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
• Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
• Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
• Familiar with using Databricks Genie.
• Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling
• Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic
• Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models

2. Databricks & PySpark Engineering

• Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
• Implement Delta Lake features-ACID transactions, schema enforcement, schema evolution, and time travel-across the data lifecycle.
• Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
• Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

3. Data Governance & Quality

• Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency in Unity Catalog.