Data Engg With Databricks - Technical Lead-Data Engg

Birlasoft

Pune District

On-site

INR 1,200,000 - 1,800,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Birlasoft is seeking an Azure Data Engineer to architect and optimize modern data platforms on Azure. The role focuses on building scalable data pipelines and analytics solutions using Azure Databricks, PySpark, ADF, and SQL.

You will ingest data from diverse sources, implement Delta Lake features, and ensure security and governance while collaborating with BI and platform teams to deliver production-ready data ecosystems.

Qualifications

  • Experience designing enterprise-grade data pipelines on Azure.
  • Strong knowledge of Azure-native services and data architecture standards.
  • Ability to optimize data pipelines for performance and reliability.

Responsibilities

  • Design end-to-end data engineering solutions using Azure Data Factory, Azure Databricks, PySpark, SQL, and other Azure-native services.
  • Architect and implement scalable and secure MDW and Lakehouse solutions leveraging Azure Data Lake Storage and Databricks.
  • Develop data models, integration patterns, and reusable frameworks aligned with enterprise standards.
  • Participate in requirement discussions, solution blueprinting, and technical feasibility assessments.
  • Build and optimize high-throughput ELT/ETL pipelines across structured, semi-structured, and unstructured data.
  • Integrate data from multiple on-premise and cloud sources, APIs, and third-party sources.
  • Implement complex transformations using PySpark with performance optimization and modular code.
  • Create orchestration workflows in ADF with pipelines, triggers, and datasets.
  • Develop Delta Lake features — ACID, schema enforcement/evolution, time travel.
  • Collaborate with BI, analytics, and business teams to deliver production-ready solutions.
  • Provide guidance and mentoring to junior engineers when required.

Skills

Azure Databricks
PySpark
Azure Data Factory
SQL

Tools

Databricks
Delta Lake

Job description

Job Description:

Area(s) of responsibility

Role Summary
We are looking for a highly skilled Azure Data Engineer with strong hands on expertise in Azure Databricks, PySpark, ADF, and SQL. The candidate will be responsible for architecting, developing, and optimizing modern data platforms and analytical solutions on Azure. The role requires strong experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Key Responsibilities

  1. Data Solution Design & Architecture
    • Design end to end data engineering solutions using Azure Data Factory, Azure Databricks, PySpark, SQL, and other Azure-native services.
    • Architect and implement scalable and secure Modern Data Warehouse (MDW) and Lakehouse solutions leveraging Azure Data Lake Storage and Databricks.
    • Develop data models, integration patterns, and reusable frameworks aligned with best practices and enterprise architecture standards.
    • Participate in requirement discussions, solution blueprinting, and technical feasibility assessments.
  2. Data Pipeline Development
    • Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
    • Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
    • Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
    • Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
  3. Databricks & PySpark Engineering
    • Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
    • Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
    • Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
    • Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.
  4. Data Governance, Quality & Security
    • Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency.
    • Ensure compliance with organizational standards for data security, encryption, access control, and data lifecycle management.
    • Create and maintain technical documentation, data flow diagrams, and operational support guides.
  5. Collaboration & Stakeholder Management
    • Collaborate closely with BI, analytics, and business teams to understand data requirements and deliver reliable, production-ready solutions.
    • Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.
    • Provide guidance and mentoring to junior engineers when required.

Requirements:

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engg With Databricks PySpark -Sr Technical Lead-Data Engg
Data Engg With Databricks PySpark -Sr Technical Lead-Data Engg

Birlasoft • Pune District

On-site
INR 1,500,000 - 2,200,000
Data Engg with Databricks - Technical Lead-Data Engg
Data Engg with Databricks - Technical Lead-Data Engg

Birlasoft • Pune District

On-site
INR 6,622,516 - 8,514,664
Data Engg with Databricks+ PySpark -Sr Technical Lead-Data Engg
Data Engg with Databricks+ PySpark -Sr Technical Lead-Data Engg

Birlasoft • Pune District

On-site
INR 1,200,000 - 1,500,000
Sr. Databricks Engineer
Sr. Databricks Engineer

ExlService Holdings, Inc. • Pune District

Hybrid
INR 3,500,000 - 6,500,000
Data Engineer-Information Technology
Data Engineer-Information Technology

Emcure Pharmaceuticals Limited • Pune District

On-site
INR 1,400,000 - 2,000,000
Databricks Engineer
Databricks Engineer

DATAECONOMY Inc • Pune District

On-site
INR 2,500,000 - 4,000,000
Senior Databricks Engineer
Senior Databricks Engineer

DATABEAT • Hyderabad

On-site
INR 1,500,000 - 2,500,000
Databricks data engineer
Databricks data engineer

NYX TECH SOLUTION • Pune District

On-site
INR 1,200,000 - 2,100,000
Databricks Architect
Databricks Architect

Infosys • Bengaluru

On-site
INR 4,000,000 - 7,000,000
Data Engineer - Databricks
Data Engineer - Databricks

Onedatasoftware • Coimbatore District

On-site
INR 1,000,000 - 1,500,000