Azure Databricks PySpark -Sr Technical Lead-Data Engg

Birlasoft; Pune District; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Birlasoft is seeking a seasoned Senior Azure Data Engineer / Sr. Technical Lead to architect and deliver enterprise data solutions on Azure. The role focuses on Databricks, PySpark, SQL, and ADF, with hands-on engineering and strong leadership across project lifecycles.

You will design end-to-end MDW and Lakehouse architectures, optimize data pipelines, and mentor engineers while collaborating with cross-functional stakeholders to build a scalable data ecosystem on Azure.

Qualifications

• Azure Databricks, PySpark, and Azure Data Factory expertise.
• Strong data modeling, MDW/Lakehouse design, and data governance.
• Hands-on engineering with scalable ETL/ELT pipelines.
• Mentor and lead engineering teams in an Agile environment.
• Ability to design end-to-end data architectures on Azure.

Responsibilities

• Lead architecture and technical leadership for data solutions.
• Build and optimize ELT/ETL pipelines with Azure services.
• Design MDW and Lakehouse patterns using Delta Lake.
• Define standards, frameworks, and CI/CD integration.
• Collaborate with cross-functional teams and stakeholders.

Skills

Azure Databricks

PySpark

ADF

SQL

Data engineering

Technical leadership

Tools

Delta Lake

CI/CD

Job description

Job Description:

Area(s) of responsibility

Role Summary
We are seeking a seasoned Senior Azure Data Engineer / Sr. Technical Lead with extensive expertise in Azure Databricks, PySpark, ADF, SQL, and modern data engineering practices. This role requires strong technical leadership, hands on engineering capabilities, and the ability to design, architect, and deliver enterprise-scale data solutions. The ideal candidate will lead complex data initiatives, mentor engineering teams, and collaborate with cross-functional stakeholders to build a robust and scalable data ecosystem on Azure.

Key Responsibilities

1. 1. Solution Architecture & Technical Leadership
   Lead the design, development, and deployment of large-scale data engineering solutions using Azure Databricks, PySpark, SQL, ADF, and Azure Data Lake.
   Architect end-to-end Modern Data Warehouse (MDW) and Lakehouse solutions, ensuring scalability, performance, security, and cost optimization.
   Define technical standards, coding best practices, reusable frameworks, and architectural guidelines for engineering teams.
   Provide technical leadership across the project lifecycle- requirements analysis, solution blueprinting, estimation, development, and deployment.
2. 2. Data Pipeline Engineering
   Build, optimize, and maintain scalable, high performance ELT/ETL pipelines to process large volumes of structured and unstructured data.
   Set up complex data ingestion frameworks, enabling seamless integration with on-premise systems, cloud services, APIs, and third-party sources.
   Ensure high availability, data reliability, and error-resilient orchestration workflows in Azure Data Factory.
3. 3. Azure Databricks & PySpark Expertise
   Design and implement advanced transformation logic using PySpark on Databricks, ensuring efficient data processing and code modularity.
   Utilize Delta Lake capabilities - ACID transactions, schema evolution, versioning, time travel to manage enterprise-grade datasets.
   Perform cluster-level tuning, optimization of shuffle operations, caching, partitioning, and job parallelization.
   Manage Databricks job pipelines, notebooks, clusters, job scheduling, and integration with CI/CD pipelines.
4. 4. Data Governance, Quality & Documentation
   Implement data quality frameworks covering validation, reconciliation, error handling, and metadata management.
   Enforce best practices for security, access control, encryption, data lineage, and auditability.
   Prepare and maintain detailed documentation: technical specification documents, interface designs, architecture diagrams, and operational runbooks.
5. 5. Stakeholder Collaboration & Team Leadership
   Collaborate with BI, analytics, business teams, and architects to convert business requirements into scalable technical solutions.
   Lead code reviews, provide mentoring and technical guidance to junior and mid-level engineers.
   Work closely with Scrum Masters and Product Owners within an Agile delivery model.