Databricks + Azure Developer

TESTQ Technologies LTD.; India; Remote; INR 4,000,000 - 7,000,000; Full time

Job summary

TESTQ Technologies LTD. in India is seeking an experienced Databricks + Azure Developer to design, build, and optimize scalable data engineering solutions on Azure.

The role requires hands-on expertise with Azure Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory, and ADLS Gen2 to develop robust ETL/ELT pipelines and lakehouse architectures. The ideal candidate will implement Medallion layers (Bronze/Silver/Gold), perform incremental loads, CDC, MERGE, and SCD Type 1/2, and optimize

Qualifications

• 9–10+ years of experience in Data Engineering or Azure cloud data development.
• Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Experience building scalable ETL/ELT pipelines using Azure Data Factory (ADF).
• Strong understanding of ADLS Gen2, Lakehouse architecture, and distributed data processing.
• Experience with CI/CD (Azure DevOps), Git, YAML pipelines, and Infrastructure-as-Code (Terraform/ARM/Bicep).
• Solid SQL skills and experience with performance tuning.
• Experience working in Agile/Scrum environments.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Build and orchestrate ETL/ELT workflows using Azure Data Factory (ADF).
• Develop ingestion and transformation pipelines for structured and semi-structured data using ADLS Gen2, Delta Lake, and Spark SQL.
• Implement Medallion architecture (Bronze/Silver/Gold layers) and incremental loads, CDC, MERGE, and SCD Type 1/2.
• Optimize Spark jobs through partitioning, caching, file-size optimization, and performance-tuning techniques.
• Configure and maintain Databricks Workflows/Jobs, clusters, and workspace environments.
• Implement data quality checks, validation, error handling, monitoring, and reconciliation processes.
• Work with Unity Catalog for governance, lineage, RBAC, and secure data access.
• Integrate Databricks with Azure services (ADLS, Azure SQL, APIs, JDBC connections).
• Implement CI/CD pipelines using Azure DevOps and Git; automate infrastructure using Terraform or ARM/Bicep.
• Troubleshoot production pipelines, perform root-cause analysis, and ensure SLA compliance.
• Collaborate with architects, analysts, BI teams, and business stakeholders in an Agile environment.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Azure Data Factory

ADLS Gen2

Lakehouse

CI/CD

Terraform/ARM/Bicep

Tools

Azure DevOps

Git

Terraform

ARM/Bicep

Databricks Administration

Job description

Job Type: Contract

Work Mode: Remote

Mode: 100% remote

Time Shift: EST Time shift

Pref: Immediate Joiner

We are looking for an experienced Databricks + Azure Developer to design, build, and optimize scalable data engineering solutions on Azure. The ideal candidate will have strong hands-on experience with Azure Databricks, PySpark, Spark SQL, Delta Lake, Azure Data Factory, ADLS Gen2, and modern data-pipeline development. The role involves building high-performance ETL/ELT pipelines, implementing Lakehouse architecture, and ensuring reliable, secure, and scalable data operations.

Key Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Build and orchestrate ETL/ELT workflows using Azure Data Factory (ADF).
• Develop ingestion and transformation pipelines for structured and semi-structured data using ADLS Gen2, Delta Lake, and Spark SQL.
• Implement Medallion architecture (Bronze/Silver/Gold layers) and incremental loads, CDC, MERGE, and SCD Type 1/2.
• Optimize Spark jobs through partitioning, caching, file-size optimization, and performance-tuning techniques.
• Configure and maintain Databricks Workflows/Jobs, clusters, and workspace environments.
• Implement data quality checks, validation, error handling, monitoring, and reconciliation processes.
• Work with Unity Catalog for governance, lineage, RBAC, and secure data access.
• Integrate Databricks with Azure services (ADLS, Azure SQL, APIs, JDBC connections).
• Implement CI/CD pipelines using Azure DevOps and Git; automate infrastructure using Terraform or ARM/Bicep.
• Troubleshoot production pipelines, perform root-cause analysis, and ensure SLA compliance.
• Collaborate with architects, analysts, BI teams, and business stakeholders in an Agile environment.

Required Qualifications

• 9–10+ years of experience in Data Engineering or Azure cloud data development.
• Strong hands-on experience with Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Experience building scalable ETL/ELT pipelines using Azure Data Factory (ADF).
• Strong understanding of ADLS Gen2, Lakehouse architecture, and distributed data processing.
• Experience with CI/CD (Azure DevOps), Git, YAML pipelines, and Infrastructure-as-Code (Terraform/ARM/Bicep).
• Solid SQL skills and experience with performance tuning.
• Experience working in Agile/Scrum environments.

Preferred Qualifications

• Experience with Databricks platform administration (clusters, jobs, workspace configuration).
• Knowledge of Unity Catalog, data governance, lineage, and RBAC.
• Experience integrating Databricks with APIs, JDBC sources, and external systems.
• Familiarity with monitoring tools (Datadog, Grafana) and cost optimization.