Azure Databricks Architect / Lead

Sightspectrum; Chennai; On-site; INR 4,000,000 - 8,000,000; Full time

Job summary

Sightspectrum in India invites an experienced Azure Databricks Lead/Architect to design and implement enterprise-scale data platforms on Azure. You will drive architecture, migration from on-prem to cloud, and optimization of pipelines, Delta Lake, and Unity Catalog.

You will lead Databricks clusters, tune performance, ensure security, and collaborate with ADLS Gen2, Azure Data Factory, Synapse, SQL, Key Vault, and DevOps to deliver reliable solutions.

Qualifications

• Extensive hands-on experience with Azure Databricks and data lakehouse architectures.
• Proven ability to architect, migrate, and implement large-scale data platforms on Azure.
• Strong skills in PySpark, Spark, Python and SQL for data processing.

Responsibilities

• Design and architect scalable Azure Databricks solutions for enterprise data platforms.
• Lead Databricks migration and implementation projects from on-premise or legacy platforms to Azure.
• Define Lakehouse architecture, data pipelines, ingestion, transformation, and processing frameworks.
• Provide technical leadership for Databricks clusters, performance tuning, scalability, and cost optimization.
• Work closely with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, Azure SQL, Key Vault, and Azure DevOps.
• Design and implement secure, reliable, and highly available data solutions.
• Establish best practices around Delta Lake, Unity Catalog, CI/CD, governance, and data security.
• Lead technical discussions with stakeholders and guide development teams.
• Troubleshoot complex performance, integration, and data engineering challenges.

Skills

Azure Databricks

Databricks architecture

PySpark

Spark

Python

SQL

Delta Lake

Unity Catalog

ADLS Gen2

Azure Data Factory

Cluster optimization

Azure security

Data engineering leadership

Tools

Terraform

Azure DevOps

Job description

We are looking for an experienced Azure Databricks Lead / Architect to drive the design, architecture, migration, implementation, and optimization of enterprise-scale data platforms on Azure.

Key Responsibilities

• Design and architect scalable Azure Databricks solutions for enterprise data platforms.
  Lead Databricks migration and implementation projects from on-premise or legacy platforms to Azure.
  Define Lakehouse architecture, data pipelines, ingestion, transformation, and processing frameworks.
  Provide technical leadership for Databricks clusters, performance tuning, scalability, and cost optimization.
  Work closely with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, Azure SQL, Key Vault, and Azure DevOps.
  Design and implement secure, reliable, and highly available data solutions.
  Establish best practices around Delta Lake, Unity Catalog, CI/CD, governance, and data security.
  Lead technical discussions with stakeholders and guide development teams.
  Troubleshoot complex performance, integration, and data engineering challenges.

Must-Have Skills

Strong hands-on experience with Azure Databricks
Expertise in Databricks architecture, migration, and implementation
Strong knowledge of PySpark / Spark / Python / SQL
Experience with Delta Lake and Unity Catalog
Strong understanding of ADLS Gen2 and Azure Data Factory
Experience in performance tuning and cluster optimization
Experience designing enterprise-scale data platforms
Strong understanding of Azure cloud architecture and security
Experience leading data engineering / architecture teams

Good to Have

Databricks certification
Experience with Azure DevOps and CI/CD
Experience with Terraform / Infrastructure as Code
Knowledge of data governance and security frameworks
Experience handling large-scale data migration projects