Databricks - Technical Specialist-Data Engg - Birlasoft

OpenTalent; Maharashtra; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

OpenTalent in Pune, India, is seeking a highly skilled Databricks Data Engineer with hands-on expertise in Databricks, PySpark, ADF, and SQL to architect, develop, and optimize modern data platforms on Azure.

You will build enterprise-grade data pipelines ingesting data from diverse sources, implement complex transformations, and deliver scalable analytics-ready models, collaborating with IT and business teams.

Responsibilities

• Build and maintain ingestion frameworks (ADF / Databricks / Spark)
• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
• Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
• Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
• Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
• Familiar with using Databicks Genie.
• Dimensional models (star/snowflake schemas)
• Fact tables, dimensions, surrogate keys, SCD handling
• Analytics-ready models
• Consistency across domains (conformed models)
• Reusability and scalability of models
• Column derivation logic changes (not just schema changes)
• Backward compatibility and impact analysis
• Define business rules
• Validate metrics and edge cases

Tools

Databricks

ADF

SQL

PySpark

Azure

Job description

Country/Region: IN

Requisition ID: 38599

Work Model:

Position Type:

Salary Range:

Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI

Title: Databricks - Technical Specialist-Data Engg

Area(s) of responsibility

Role Summary We are looking for a highly skilled Databricks Data Engineer with strong hands on expertise in Databricks, PySpark, ADF, and SQL. The candidate will be responsible for architecting, developing, and optimizing modern data platforms and analytical solutions on Azure. The role requires strong experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Key Responsibilities

• Build and maintain ingestion frameworks (ADF / Databricks / Spark)
• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
• Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
• Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
• Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
• Familiar with using Databicks Genie.
• Dimensional models (star/snowflake schemas)
• Fact tables, dimensions, surrogate keys, SCD handling
• Analytics-ready models
• Consistent KPI definitions and business logic
• Consistency across domains (common dimensions, conformed models)
• Reusability and scalability of models
• Column derivation logic changes (not just schema changes)
• Backward compatibility and impact analysis
• Define business rules
• Validate metrics and edge cases

Collaboration & Stakeholder Management

Collaborate closely with customer IT and business teams to understand data requirements and deliver reliable, production-ready solutions. Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives. Provide guidance and mentoring to junior engineers when required.