Databrick Engineer

Workforce Solution; Thane; On-site; INR 1,800,000 - 2,500,000; Full time

Job summary

Workforce Solution in India (Thane) seeks a Data Engineer to design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark. You will build and optimize ETL/ELT processes across multiple data sources.

You will implement Delta Lake architecture for reliable processing, tune Databricks jobs, and support production deployments with L2/L3 troubleshooting while ensuring data governance and security.

Qualifications

• Experience designing data pipelines with Azure Databricks and PySpark.
• Ability to implement Delta Lake for reliable data processing.
• Experience with Azure Data Factory for data ingestion.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and optimize ETL/ELT processes for ingesting data from multiple source systems.
• Develop data transformation logic and implement business rules using Spark frameworks.
• Work closely with stakeholders, analysts, and architects to understand data requirements.
• Implement and manage Delta Lake architecture for reliable, high-performance processing.
• Optimize Databricks jobs, notebooks, clusters, and Spark code for performance and cost.
• Develop and support data ingestion frameworks using Azure Data Factory and other tools.
• Monitor production workloads, troubleshoot failures, and ensure data availability.
• Implement data quality checks, validations, and reconciliation processes.
• Ensure adherence to data governance, security, and compliance standards.
• Participate in code reviews, technical design discussions, and best practice implementation.
• Support production deployments and provide L2/L3 support for critical data pipelines.

Skills

Azure Databricks

PySpark

Delta Lake

ETL/ELT

Azure Data Factory

Spark tuning

Data governance

Job description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and optimize ETL/ELT processes for ingesting data from multiple source systems.
• Develop data transformation logic and implement business rules using Spark frameworks.
• Work closely with business stakeholders, data analysts, and architects to understand data requirements.
• Implement and manage Delta Lake architecture for reliable and high-performance data processing.
• Optimize Databricks jobs, notebooks, clusters, and Spark code for performance and cost efficiency.
• Develop and support data ingestion frameworks using Azure Data Factory (ADF) and other integration tools.
• Monitor production workloads, troubleshoot failures, and ensure data availability.
• Implement data quality checks, validations, and reconciliation processes.
• Ensure adherence to data governance, security, and compliance standards.
• Participate in code reviews, technical design discussions, and best practice implementation.
• Support production deployments and provide L2/L3 support for critical data pipelines.