Databricks - Technical Specialist-Data Engg

Birlasoft Limited; Pune District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Birlasoft Limited is seeking a Databricks Data Engineer in Pune to architect, develop, and optimize modern data platforms on Azure. You will build robust data pipelines, implement Bronze to Silver transformations, and ensure data quality across enterprise datasets.

The role involves collaborating with IT and business teams to deliver scalable analytics solutions. Qualifications include hands-on Databricks, PySpark, ADF, and SQL, plus experience with Delta Lake and data modeling.

Qualifications

• Hands-on expertise with Databricks, PySpark, ADF and SQL.
• Experience architecting data platforms on Azure.
• Proven ability to build enterprise-grade data pipelines.

Responsibilities

• Build and maintain ingestion frameworks (ADF / Databricks / Spark).
• Implement Bronze -> Silver transformations aligned to architecture.
• Architect data quality checks, schema validation, and contract rules.
• Build and optimize robust ELT/ETL pipelines for structured, semi-structured, and unstructured data.
• Integrate data from on-premise and cloud systems, APIs, and third-party sources.
• Implement complex transformations using PySpark for performance and modularity.
• Build orchestration workflows in ADF: pipelines, triggers, linked services, and datasets.
• Familiar with using Databricks Genie.
• Design and implement curated data models in Gold layer aligned to business use cases.
• Build dimensional models, star/snowflake schemas, facts, dimensions, SCDs.
• Translate Silver datasets into analytics-ready models with KPI logic.
• Ensure cross-domain consistency and model reusability and scalability.
• Develop scalable PySpark scripts on Databricks with caching, partitioning, Delta Lake features.
• Apply ACID, schema enforcement/evolution, and time travel in Delta Lake.
• Performance tuning for clusters, joins, shuffles, and parallelization.
• Collaborate with platform teams to manage clusters, jobs, notebooks, CI/CD.
• Implement data quality checks and Unity Catalog governance.
• Enforce Unity Catalog standards, metadata policies, and access controls.
• Partner with data stewards to define rules and validate metrics.
• Collaborate with IT and business teams to deliver production-ready solutions.
• Provide guidance and mentoring to junior engineers.

Skills

Databricks

PySpark

Azure

SQL

Data modeling

Delta Lake

CI/CD

Tools

Databricks Genie

ADF

Spark

Job description

Area(s) of responsibility Role Summary We are looking for a highly skilled Databricks Data Engineer with strong hands on expertise in Databricks, PySpark, ADF, and SQL. The candidate will be responsible for architecting, developing, and optimizing modern data platforms and analytical solutions on Azure. The role requires strong experience building enterprise-grade data pipelines, enabling ingestion from diverse sources, implementing complex transformations, and supporting scalable analytics initiatives.

Key Responsibilities

Data Engineering Execution

• Build and maintain ingestion frameworks (ADF / Databricks / Spark) Implement Bronze -> Silver transformations aligned to architecture Architect data quality checks, schema validation, and contract rules
• Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi structured, and unstructured data.
• Integrate data from multiple on premise and cloud based systems, APIs, and third-party sources.
• Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
• Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
• Familiar with using Databicks Genie.

Design and implement curated data models in Gold layer aligned to business use cases

• Build:
  • Dimensional models (star/snowflake schemas)
  • Fact tables, dimensions, surrogate keys, SCD handling
• Translate Silver datasets into:
  • Analytics-ready models
  • Consistent KPI definitions and business logic
• Ensure:
  • Consistency across domains (common dimensions, conformed models)
  • Reusability and scalability of models

Databricks & PySpark Engineering

• Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
• Implement Delta Lake features ACID transactions, schema enforcement, schema evolution, and time travel across the data lifecycle.
• Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
• Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.

Data Governance & Quality

• Implement data quality checks, audit mechanisms, and validation frameworks to ensure data accuracy and consistency in Unity Catalog.
• Enforce:
  • Unity Catalog standards
  • naming conventions, metadata policies
  • access controls (RBAC/ABAC)
  • Handle changes such as:
    • Column derivation logic changes (not just schema changes)
    • Backward compatibility and impact analysis
• Partner with business / data stewards to:
  • Define business rules
  • Validate metrics and edge cases

Collaboration & Stakeholder Management

• Collaborate closely with customer IT and business teams to understand data requirements and deliver reliable, production-ready solutions.
• Work with architects, product owners, and cross-functional engineering teams to align technical delivery with business objectives.
• Provide guidance and mentoring to junior engineers when required.