Senior Databricks Architect

Zorba AI; Pune District; Presencial; INR 4.000.000 - 12.000.000; Jornada completa

Descripción de la vacante

Zorba AI is seeking an experienced Senior Databricks Architect to lead enterprise-scale Lakehouse architecture, modernization, governance, and migration efforts. You will define standards for Databricks, Spark, Delta Lake, and Unity Catalog, while driving performance optimization and cost controls across AWS/Azure/GCP.

The role requires hands-on expertise, strong architecture leadership, and the ability to mentor engineering teams in building AI-ready data foundations and robust cloud data

Formación

• 10–15 years of IT experience with data engineering and architecture.
• Strong hands-on Databricks and Spark expertise.
• Experience with Unity Catalog, Delta Lake and Medallion architecture.
• Cloud data platform design across AWS, Azure or GCP.
• Proven leadership, stakeholder management and mentoring.

Responsabilidades

• Design and lead enterprise-scale Databricks Lakehouse architecture including workspace strategy and Unity Catalog.
• Define standards for Databricks, PySpark, Spark SQL, Delta Lake, DLT/Lakehouse, and Workflows.
• Lead modernization and migration from legacy warehouses to Databricks.
• Drive Spark performance optimization: AQE, partitioning, caching, and Photon.
• Establish governance, security, lineage, and data quality frameworks with Unity Catalog.
• Set CI/CD, DevOps, and IaC standards for Databricks platforms.
• Define reliability, monitoring, observability, and disaster recovery practices.
• Lead cost optimization through autoscaling, FinOps, and serverless strategies.
• Engage stakeholders, conduct architecture workshops, and mentor teams.

Conocimientos

Databricks Lakehouse

Apache Spark

PySpark

Delta Lake

Unity Catalog

CI/CD

Terraform/IaC

DevOps

Data governance

Data migration

Cloud platforms

Descripción del empleo

We are looking for an experienced Senior Databricks Architect with 10–15 years of IT experience to lead enterprise-scale Lakehouse architecture, platform modernization, data governance, and migration initiatives. The candidate should have strong hands‑on expertise in Databricks, Apache Spark, PySpark, Delta Lake, and Unity Catalog, along with the ability to define architecture strategies, engineering standards, performance optimization, and cloud data platform roadmaps.

Key Responsibilities

• Design and lead enterprise-scale Databricks Lakehouse architecture, including workspace strategy, Unity Catalog, storage, compute, and Medallion architecture.
• Define architecture standards and best practices for Databricks, PySpark, Spark SQL, Delta Lake, DLT/Lakeflow, Workflows, and Auto Loader.
• Lead large-scale data platform modernization and migration from legacy data warehouses, Hadoop, and ETL platforms to Databricks.
• Drive Spark performance optimization, including query tuning, AQE, partitioning, caching, shuffle optimization, and Photon.
• Design enterprise data governance, security, access controls, lineage, data quality, and Delta Sharing frameworks using Unity Catalog.
• Establish CI/CD, DevOps, Infrastructure as Code (Terraform), automated testing, and deployment standards for Databricks platforms.
• Define platform reliability, monitoring, observability, disaster recovery, and production support standards.
• Drive Databricks cost optimization through cluster policies, autoscaling, compute sizing, serverless adoption, and FinOps practices.
• Build AI‑ready data foundations and demonstrate awareness of RAG, Vector Search, MLflow, Model Serving, and AI governance.
• Lead architecture workshops, engage with business and technical stakeholders, define technology roadmaps, and mentor engineering teams.

Mandatory Skills

• 10–15 years of overall IT experience with significant experience in Data Engineering and Architecture.
• Strong expertise in Databricks Lakehouse Architecture and Apache Spark.
• Hands‑on experience with PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
• Strong knowledge of Unity Catalog, Data Governance, Security, and Data Lineage.
• Experience designing enterprise-scale cloud data platforms on AWS, Azure, or GCP.
• Expertise in data migration, modernization, performance tuning, and platform optimization.
• Experience with CI/CD, DevOps, Terraform/IaC, and production‑grade data platform engineering.
• Strong architecture leadership, stakeholder management, solution design, and technical mentoring skills.

Good to Have

• Experience with Lakeflow Declarative Pipelines / Delta Live Tables (DLT), Auto Loader, and Databricks Workflows.
• Knowledge of Structured Streaming, CDC, SCD, and Delta Sharing.
• Exposure to Databricks Asset Bundles, system tables, Lakehouse Federation, and serverless compute.
• Awareness of MLflow, Vector Search, RAG, and AI/ML platform integration.
• Experience In Enterprise FinOps, Chargeback/showback, And Cloud Cost Governance.

Skills: sql,tuning,databricks,migration,spark sql,apache spark,pyspark,lakehouse,unity,aws,ci/cd