Azure Databricks Developer

Birlasoft

Pune District

On-site

INR 900,000 - 1,500,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Birlasoft is seeking a data engineering professional to build and optimize ingestion and transformation pipelines on Databricks and ADF. You will design Bronze-to-Silver pipelines, enforce schema validation and contract rules, and ensure high-throughput data processing across diverse sources.

The role emphasizes scalable PySpark implementations, Delta Lake capabilities, and orchestration via ADF, with an eye on performance tuning and reusable data models.

Qualifications

  • Strong experience building scalable ELT/ETL pipelines.
  • Expertise in data modeling, schema design, and dimensional models.
  • Proficient in orchestrating data workflows and ensuring data quality across domains.

Responsibilities

  • Build and maintain ingestion frameworks (ADF / Databricks / Spark).
  • Implement Bronze -> Silver transformations aligned to architecture.
  • Architect data quality checks, schema validation, and contract rules.
  • Build and optimize robust ETL/ELT pipelines for structured, semi-structured, and unstructured data.
  • Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

Skills

PySpark
Databricks
Spark
ETL/ELT
Data modeling
CI/CD
Data quality

Tools

ADF
Databricks
Delta Lake
Spark
Python

Job description

  • Build and maintain ingestion frameworks (ADF / Databricks / Spark)
  • Implement Bronze -> Silver transformations aligned to architecture
  • Architect data quality checks, schema validation, and contract rules
  • Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
  • Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
  • Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
  • Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
  • Familiar with using Databricks Genie.
  • Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling
  • Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions and business logic
  • Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models
  • Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
  • Implement Delta Lake features—ACID transactions, schema enforcement, schema evolution, and time travel—across the data lifecycle.
  • Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
  • Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer
Senior Data Engineer

Bridgestone Americas, Inc. • Bengaluru Urban

On-site
INR 1,500,000 - 2,300,000
Senior Databricks Engineer
Senior Databricks Engineer

Aubrant Digital • Hyderabad

On-site
INR 3,000,000 - 6,000,000
Databricks Tech Lead
Databricks Tech Lead

Fa Ewjt Saasfaprod1 • Bengaluru

On-site
INR 2,500,000 - 4,500,000
Azure Databricks Engineer
Azure Databricks Engineer

Turnkeylearning • Hyderabad

On-site
INR 1,000,000 - 2,000,000
Senior Data Brick Engineer
Senior Data Brick Engineer

Zenithive Engineering • Ahmedabad District

On-site
INR 1,200,000 - 2,400,000
Databricks Engineer
Databricks Engineer

Infosys • Bengaluru

On-site
INR 2,800,000 - 4,400,000
Data Engineer | PySpark | Databricks | Azure Data Factory
Data Engineer | PySpark | Databricks | Azure Data Factory

Infosys • Hyderabad, Pune District, Bengaluru

Hybrid
INR 1,200,000 - 2,400,000
Azure Databricks Data Engineer
Azure Databricks Data Engineer

WTW group • Kolkata District

On-site
INR 1,800,000 - 3,400,000
Azure Databricks Data Engineer
Azure Databricks Data Engineer

Infosys • Bengaluru

On-site
INR 1,800,000 - 3,200,000
Azure Databricks Data Engineer
Azure Databricks Data Engineer

Infosys • Hyderabad, Pune District, Bengaluru

On-site
INR 850,000 - 1,200,000