Tech Lead – Databricks / Data Platform

Palo Alto Innovation Labs; Pune District; Remote; INR 4,000,000 - 6,500,000; Full time

Job summary

Palo Alto Innovation Labs is seeking a Tech Lead – Databricks / Data Platform to guide the design, development, and optimization of a scalable, secure cloud-based data platform. You will lead teams across Databricks, Azure, CI/CD, IaC, and data governance to deliver enterprise-grade data solutions.

The role requires hands-on leadership, collaboration with data engineers, analytics engineers, architects, security, and cloud teams to ensure reliable, compliant data platforms.

Qualifications

• 10–12 years of experience in data, cloud, or DevOps engineering.
• Hands-on Databricks production experience with workspace and clusters.
• Strong Microsoft Azure expertise and cloud-native data platforms.
• CI/CD and Git-based development workflows (Azure DevOps, GitHub Actions, GitLab CI).
• In-depth Terraform / IaC and cloud infrastructure automation.
• Hands-on Apache Airflow (or managed Airflow) for data pipelines.
• Programming/scripting in Python, Bash, or PowerShell.
• Experience implementing automated testing, validation, and data quality gates.
• Experience with monitoring, logging, and incident management in production.
• Strong understanding of cloud security, IAM, RBAC, and data governance.
• Experience with FinOps and cloud cost optimization.

Responsibilities

• Lead architecture, design, and implementation of Databricks-based data platforms.
• Design and implement robust CI/CD pipelines for data ingestion and transformation.
• Lead end-to-end orchestration of Databricks Jobs and Workflows.
• Define and implement automated testing and quality gates in CI/CD pipelines.
• Drive Infrastructure as Code practices using Terraform for Databricks/Azure infra.
• Automate Databricks workspace, cluster provisioning, and environment management.
• Establish IAM across Databricks and Azure including RBAC and access controls.
• Collaborate with EDP Architect and Security on Unity Catalog and data governance.
• Develop reusable Terraform modules and Airflow DAG patterns.
• Govern code promotion and release management across environments.
• Lead end-to-end orchestration using managed Airflow with reliable scheduling.
• Define and track platform SLAs/SLOs and manage incident triage and RCAs.
• Implement FinOps best practices for cloud cost optimization.
• Mentor engineers and promote engineering standards and patterns.
• Partner with Data Eng, Analytics, Security, Infra to deliver enterprise data solutions.
• Contribute to cloud infra, cybersecurity, disaster recovery, monitoring, and ops readiness.
• Drive performance improvements for production data workloads.

Skills

Databricks

Azure

CI/CD

Terraform

Airflow

Python

IAM

FinOps

Communication

Education

Bachelor's degree in Computer Science/Engineering or related

Tools

Databricks

Azure

Terraform

Airflow

dbt

SQL

Delta Lake

Unity Catalog

Job description

Role : Tech Lead – Databricks / DataPlatform
Location: Remote
Employment Type: Full-Time
Experience: 10–12 Years

JOBSUMMARY

Weare looking for an experienced Tech Lead – Databricks / Data Platform tolead the design, development, and optimization of a scalable, secure, andreliable cloud-based data platform. The role will provide technical leadershipacross Databricks, Azure, CI/CD, Infrastructure as Code, orchestration, datasecurity, FinOps, and platform automation.

Theideal candidate will be hands-on, technically strong, and experienced inleading engineering teams while partnering closely with Data Engineers,Analytics Engineers, Architects, Security, and Cloud teams to deliverenterprise-grade data solutions.

KEYRESPONSIBILITIES

• Leadthe architecture, design, and implementation of Databricks-based dataplatforms and cloud-native data pipelines.
  
• Designand implement robust CI/CD pipelines for data ingestion,transformation, dbt, SQL, notebooks, and Databricks workloads.
  
• Leadend-to-end orchestration of Databricks Jobs and Workflows,including ingestion, transformation, data quality checks, anddependencies.
  
• Defineand implement automated testing and quality gates within CI/CDpipelines, including schema, data model, and contract validation.
  
• Drive Infrastructureas Code (IaC) practices using Terraform and reusable modules forDatabricks and Azure infrastructure.
  
• AutomateDatabricks platform operations, including workspace and clusterprovisioning, runtime and library management, job deployment,configuration, and environment management.
  
• Establishand maintain Identity and Access Management (IAM) across Databricksand Azure, including RBAC, TBAC, service principals, groups, roles, andworkspace/cluster/table-level access controls.
  
• Workclosely with the EDP Architect and Security teams to implement secure andscalable Unity Catalog and data governance patterns.
  
• Developreusable Terraform modules, Databricks job templates, Airflow DAGpatterns, and engineering frameworks to accelerate onboarding of newprojects.
  
• Defineand govern the code promotion and release management process acrossdevelopment, QA, and production environments.
  
• Leadend-to-end orchestration using managed Airflow, ensuring reliablescheduling, dependency management, monitoring, and recovery.
  
• Defineand track platform SLAs, SLOs, and SLIs, and lead incident triage,root cause analysis, and corrective actions.
  
• Implement FinOps best practices for monitoring, optimizing, and allocatingDatabricks and cloud infrastructure costs.
  
• Continuouslyevaluate and improve platform tooling, architecture, reliability,security, performance, and developer productivity.
  
• Providetechnical guidance and mentorship to engineers and establish engineeringbest practices, standards, and reusable patterns.
  
• Partnerwith Data Engineering, Analytics, Architecture, Security, andInfrastructure teams to ensure seamless delivery of enterprise datasolutions.
  
• Contributeto cloud infrastructure, cybersecurity, disaster recovery, monitoring,logging, and operational readiness.
  
• Driveperformance optimization and reliability improvements for production dataworkloads.
  

REQUIREDQUALIFICATIONS

• 10–12years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE,or related technical disciplines.
  
• Stronghands-on experience with Databricks in production environments,including workspace and cluster management, Jobs/Workflows, Unity Catalog,and integration with orchestration platforms.
  
• Strongexperience with Microsoft Azure and cloud-native data platforms.
  
• Strongknowledge of CI/CD and Git-based development workflows, using toolssuch as Azure DevOps, GitHub Actions, GitLab CI, or similar.
  
• Strongexperience with Terraform / Infrastructure as Code and cloudinfrastructure automation.
  
• Hands-onexperience with Apache Airflow or managed Airflow for data pipelineorchestration.
  
• Strongprogramming/scripting skills in Python, Bash, or PowerShell.
  
• Experienceimplementing automated testing, validation, and quality gates for datapipelines and data models.
  
• Experiencewith production workload management, including monitoring, logging,troubleshooting, performance tuning, and incident management.
  
• Strongunderstanding of cloud security, IAM, RBAC, data access controls, andgovernance.
  
• Experiencewith FinOps, cloud cost optimization, and resource utilizationmonitoring.
  
• Abilityto provide technical leadership, mentor engineers, and drive engineeringstandards across teams.
  
• Strongcommunication and stakeholder management skills with the ability to workeffectively with technical and business stakeholders.
  

PREFERREDQUALIFICATIONS

• Bachelor’sdegree in Computer Science, Information Technology, Engineering, or arelated field, or equivalent professional experience.
  
• Experiencewith dbt, SQL, Delta Lake, and modern data engineering practices.
  
• Experiencewith Unity Catalog and enterprise data governance.
  
• Experiencedesigning reusable platform frameworks and accelerators.
  
• Experiencein CPG, retail, manufacturing, or distribution environments.
  
• Experiencewith disaster recovery, business continuity, and highly available cloudarchitectures.
  
• Databricksor Azure certifications are a plus.