Tech Lead – Databricks / Data Platform

Palo Alto Labs; Pune District; Remote; INR 4,500,000 - 7,500,000; Full time

Job summary

Palo Alto Labs is seeking a Tech Lead – Databricks / Data Platform to design, develop, and optimize a scalable cloud-based data platform. You will provide technical leadership across Databricks, Azure, CI/CD, IaC, and platform automation, partnering with Data Engineers, Analytics Engineers, Architects, Security, and Cloud teams to deliver enterprise-grade data solutions.

The ideal candidate will be hands-on, technically strong, and experienced in leading engineering teams while aligning with

Qualifications

• 10–12 years of experience in data engineering, cloud engineering, DevOps, or platform engineering.
• Hands-on Databricks expertise in production environments with data pipelines.
• Strong knowledge of Azure cloud and integration with data platforms.
• Proficient in CI/CD using Azure DevOps, GitHub Actions, GitLab CI, or similar.
• Infrastructure as Code using Terraform for Databricks and Azure resources.
• Experience with Apache Airflow for data pipeline orchestration.
• Programming/scripting skills in Python, Bash, or PowerShell.
• Experience implementing automated testing, validation, and quality gates for data pipelines.

Responsibilities

• Lead architecture and implementation of Databricks-based data platforms and pipelines.
• Design robust CI/CD pipelines for ingestion, transformation, and Databricks workloads.
• Orchestrate Databricks Jobs/Workflows end-to-end with data quality checks.
• Define automated testing and quality gates within CI/CD.
• Drive IaC practices using Terraform for Databricks/Azure infra.
• Automate Databricks platform operations: workspace, clusters, deployments.
• Manage IAM across Databricks and Azure: RBAC, service principals, access controls.
• Collaborate with EDP Architect & Security for Unity Catalog and governance patterns.
• Develop reusable platform frameworks to accelerate new projects.
• Define and govern code promotion and release management across environments.
• Lead end-to-end orchestration using managed Airflow with monitoring and recovery.
• Define and monitor platform SLAs, SLOs, and SLIs; drive incident response.
• Implement FinOps for monitoring and optimizing cloud costs.
• Improve tooling, reliability, security, performance, and developer productivity.

Skills

Databricks production

Azure cloud

CI/CD workflows

Terraform / IaC

Apache Airflow

Python / Bash / PowerShell

Data governance / IAM

FinOps / cloud cost

Leadership / mentoring

SRE / reliability

Education

Bachelor’s degree in CS / IT / Engineering

Tools

Azure DevOps

GitHub Actions

GitLab CI

Terraform

Unity Catalog

Airflow (managed or self-hosted)

Job description

Role : Tech Lead – Databricks / DataPlatform
Location: Remote
Employment Type: Full-Time
Experience: 10–12 Years

JOBSUMMARY

Weare looking for an experienced Tech Lead – Databricks / Data Platform tolead the design, development, and optimization of a scalable, secure, andreliable cloud-based data platform. The role will provide technical leadershipacross Databricks, Azure, CI/CD, Infrastructure as Code, orchestration, datasecurity, FinOps, and platform automation.

Theideal candidate will be hands-on, technically strong, and experienced inleading engineering teams while partnering closely with Data Engineers,Analytics Engineers, Architects, Security, and Cloud teams to deliverenterprise-grade data solutions.

KEYRESPONSIBILITIES

• Leadthe architecture, design, and implementation of Databricks-based dataplatforms and cloud-native data pipelines.
  
• Designand implement robust CI/CD pipelines for data ingestion,transformation, dbt, SQL, notebooks, and Databricks workloads.
  
• Leadend-to-end orchestration of Databricks Jobs and Workflows,including ingestion, transformation, data quality checks, anddependencies.
  
• Defineand implement automated testing and quality gates within CI/CDpipelines, including schema, data model, and contract validation.
  
• Drive Infrastructureas Code (IaC) practices using Terraform and reusable modules forDatabricks and Azure infrastructure.
  
• AutomateDatabricks platform operations, including workspace and clusterprovisioning, runtime and library management, job deployment,configuration, and environment management.
  
• Establishand maintain Identity and Access Management (IAM) across Databricksand Azure, including RBAC, TBAC, service principals, groups, roles, andworkspace/cluster/table-level access controls.
  
• Workclosely with the EDP Architect and Security teams to implement secure andscalable Unity Catalog and data governance patterns.
  
• Developreusable Terraform modules, Databricks job templates, Airflow DAGpatterns, and engineering frameworks to accelerate onboarding of newprojects.
  
• Defineand govern the code promotion and release management process acrossdevelopment, QA, and production environments.
  
• Leadend-to-end orchestration using managed Airflow, ensuring reliablescheduling, dependency management, monitoring, and recovery.
  
• Defineand track platform SLAs, SLOs, and SLIs, and lead incident triage,root cause analysis, and corrective actions.
  
• Implement FinOps best practices for monitoring, optimizing, and allocatingDatabricks and cloud infrastructure costs.
  
• Continuouslyevaluate and improve platform tooling, architecture, reliability,security, performance, and developer productivity.
  
• Providetechnical guidance and mentorship to engineers and establish engineeringbest practices, standards, and reusable patterns.
  
• Partnerwith Data Engineering, Analytics, Architecture, Security, andInfrastructure teams to ensure seamless delivery of enterprise datasolutions.
  
• Contributeto cloud infrastructure, cybersecurity, disaster recovery, monitoring,logging, and operational readiness.
  
• Driveperformance optimization and reliability improvements for production dataworkloads.
  

REQUIREDQUALIFICATIONS

• 10–12years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE,or related technical disciplines.
  
• Stronghands-on experience with Databricks in production environments,including workspace and cluster management, Jobs/Workflows, Unity Catalog,and integration with orchestration platforms.
  
• Strongexperience with Microsoft Azure and cloud-native data platforms.
  
• Strongknowledge of CI/CD and Git-based development workflows, using toolssuch as Azure DevOps, GitHub Actions, GitLab CI, or similar.
  
• Strongexperience with Terraform / Infrastructure as Code and cloudinfrastructure automation.
  
• Hands-onexperience with Apache Airflow or managed Airflow for data pipelineorchestration.
  
• Strongprogramming/scripting skills in Python, Bash, or PowerShell.
  
• Experienceimplementing automated testing, validation, and quality gates for datapipelines and data models.
  
• Experiencewith production workload management, including monitoring, logging,troubleshooting, performance tuning, and incident management.
  
• Strongunderstanding of cloud security, IAM, RBAC, data access controls, andgovernance.
  
• Experiencewith FinOps, cloud cost optimization, and resource utilizationmonitoring.
  
• Abilityto provide technical leadership, mentor engineers, and drive engineeringstandards across teams.
  
• Strongcommunication and stakeholder management skills with the ability to workeffectively with technical and business stakeholders.
  

PREFERREDQUALIFICATIONS

• Bachelor’sdegree in Computer Science, Information Technology, Engineering, or arelated field, or equivalent professional experience.
  
• Experiencewith dbt, SQL, Delta Lake, and modern data engineering practices.
  
• Experiencewith Unity Catalog and enterprise data governance.
  
• Experiencedesigning reusable platform frameworks and accelerators.
  
• Experiencein CPG, retail, manufacturing, or distribution environments.
  
• Experiencewith disaster recovery, business continuity, and highly available cloudarchitectures.
  
• Databricksor Azure certifications are a plus.