Databricks + Azure Developer - Bangalore(Hybrid)

IT Minds LLC; Bengaluru; Hybrid; INR 4,000,000 - 7,000,000; Full time

Job summary

IT Minds LLC in Bengaluru (Hybrid) seeks an experienced Databricks + Azure Developer to design, build, and optimize scalable data pipelines on Azure. You will implement Medallion architecture, develop end-to-end ETL/ELT processes, and govern data access.

You will collaborate with architects, BI teams, and stakeholders in an Agile setup, ensuring reliable, secure, and scalable data operations on Azure.

Qualifications

• 10+ years of experience in Data Engineering or Azure cloud data development.
• Strong hands on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Experience building scalable ETL/ELT pipelines using Azure Data Factory (ADF).
• Strong SQL skills and performance tuning.
• Experience with CI/CD (Azure DevOps), Git, YAML pipelines, and Infrastructure as Code (Terraform/ARM/Bicep).
• Experience working in Agile/Scrum environments.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Build and orchestrate ETL/ELT workflows using Azure Data Factory (ADF).
• Develop ingestion and transformation pipelines for structured and semi-structured data using ADLS Gen2, Delta Lake, and Spark SQL.
• Implement Medallion architecture (Bronze/Silver/Gold) and incremental loads, CDC, MERGE, and SCD Type 1/2.
• Optimize Spark jobs through partitioning, caching, file size optimization, and performance tuning techniques.
• Configure and maintain Databricks Workflows/Jobs, clusters, and workspace environments.
• Implement data quality checks, validation, error handling, monitoring, and reconciliation processes.
• Work with Unity Catalog for governance, lineage, RBAC, and secure data access.
• Integrate Databricks with Azure services (ADLS, Azure SQL, APIs, JDBC connections).
• Implement CI/CD pipelines using Azure DevOps and Git; automate infrastructure using Terraform or ARM/Bicep.
• Troubleshoot production pipelines, root cause analysis, and ensure SLA compliance.
• Collaborate with architects, analysts, BI teams, and business stakeholders in an Agile environment.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Azure Data Factory

ADLS Gen2

Data pipelines

CI/CD

SQL

Agile/Scrum

Tools

Azure DevOps

Git

Terraform

ARM/Bicep

Job description

Location - Hybrid in Bangalore, ballendur location

We are looking for an experienced Databricks + Azure Developer to design, build, and optimize scalable data engineering solutions on Azure. The ideal candidate will have strong hands on experience with Azure Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory, ADLS Gen2, and modern data pipeline development. The role involves building high performance ETL/ELT pipelines, implementing Lakehouse architecture, and ensuring reliable, secure, and scalable data operations.

Key Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Build and orchestrate ETL/ELT workflows using Azure Data Factory (ADF).
• Develop ingestion and transformation pipelines for structured and semi structured data using ADLS Gen2, Delta Lake, and Spark SQL.
• Implement Medallion architecture (Bronze/Silver/Gold layers) and incremental loads, CDC, MERGE, and SCD Type 1/2.
• Optimize Spark jobs through partitioning, caching, file size optimization, and performance tuning techniques.
• Configure and maintain Databricks Workflows/Jobs, clusters, and workspace environments.
• Implement data quality checks, validation, error handling, monitoring, and reconciliation processes.
• Work with Unity Catalog for governance, lineage, RBAC, and secure data access.
• Integrate Databricks with Azure services (ADLS, Azure SQL, APIs, JDBC connections).
• Implement CI/CD pipelines using Azure DevOps and Git; automate infrastructure using Terraform or ARM/Bicep.
• Troubleshoot production pipelines, perform root cause analysis, and ensure SLA compliance.
• Collaborate with architects, analysts, BI teams, and business stakeholders in an Agile environment.

Required Qualifications

• 10+ years of experience in Data Engineering or Azure cloud data development.
• Strong hands on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Experience building scalable ETL/ELT pipelines using Azure Data Factory (ADF).
• Strong understanding of ADLS Gen2, Lakehouse architecture, and distributed data processing.
• Experience with CI/CD (Azure DevOps), Git, YAML pipelines, and Infrastructure as Code (Terraform/ARM/Bicep).
• Solid SQL Skills And Experience With Performance Tuning.
• Experience working in Agile/Scrum environments.

Preferred Qualifications

• Experience with Databricks platform administration (clusters, jobs, workspace configuration).
• Knowledge of Unity Catalog, data governance, lineage, and RBAC.
• Experience integrating Databricks with APIs, JDBC sources, and external systems.
• Familiarity with monitoring tools (Datadog, Grafana) and cost optimization.