Senior Databricks Architect

Zorba AI; Bengaluru; Presencial; INR 4.000.000 - 7.000.000; Jornada completa

Descripción de la vacante

Zorba AI in Bengaluru, India is seeking an experienced Senior Databricks Architect to lead enterprise-scale Lakehouse initiatives. You will shape workspace strategies, governance, and modernization of data platforms using Databricks, Unity Catalog, Delta Lake, and Medallion architecture.

You will drive performance tuning, cost optimization, and CI/CD practices, mentor engineering teams, and collaborate with business stakeholders to define roadmaps for scalable data platforms across AWS/Azure/GCP.

Formación

• 10–15 years of IT experience with data engineering and architecture.

Responsabilidades

• Design and lead enterprise-scale Databricks Lakehouse architecture, including workspace strategy and Unity Catalog.

Conocimientos

Databricks Lakehouse

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

Data Governance

Cloud Platforms (AWS/Azure/GCP)

CI/CD/DevOps

Terraform/IaC

Platform Architecture Leadership

Herramientas

Delta Live Tables (DLT)

Databricks Workflows

Auto Loader

Descripción del empleo

We are looking for an experienced Senior Databricks Architect with 10–15 years of IT experience to lead enterprise-scale Lakehouse architecture, platform modernization, data governance, and migration initiatives. The candidate should have strong hands‑on expertise in Databricks, Apache Spark, PySpark, Delta Lake, and Unity Catalog, along with the ability to define architecture strategies, engineering standards, performance optimization, and cloud data platform roadmaps.

Key Responsibilities

• Design and lead enterprise-scale Databricks Lakehouse architecture, including workspace strategy, Unity Catalog, storage, compute, and Medallion architecture.
• Define architecture standards and best practices for Databricks, PySpark, Spark SQL, Delta Lake, DLT/Lakeflow, Workflows, and Auto Loader.
• Lead large-scale data platform modernization and migration from legacy data warehouses, Hadoop, and ETL platforms to Databricks.
• Drive Spark performance optimization, including query tuning, AQE, partitioning, caching, shuffle optimization, and Photon.
• Design enterprise data governance, security, access controls, lineage, data quality, and Delta Sharing frameworks using Unity Catalog.
• Establish CI/CD, DevOps, Infrastructure as Code (Terraform), automated testing, and deployment standards for Databricks platforms.
• Define platform reliability, monitoring, observability, disaster recovery, and production support standards.
• Drive Databricks cost optimization through cluster policies, autoscaling, compute sizing, serverless adoption, and FinOps practices.
• Build AI‑ready data foundations and demonstrate awareness of RAG, Vector Search, MLflow, Model Serving, and AI governance.
• Lead architecture workshops, engage with business and technical stakeholders, define technology roadmaps, and mentor engineering teams.

Mandatory Skills

• 10–15 years of overall IT experience with significant experience in Data Engineering and Architecture.
• Strong expertise in Databricks Lakehouse Architecture and Apache Spark.
• Hands‑on experience with PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
• Strong knowledge of Unity Catalog, Data Governance, Security, and Data Lineage.
• Experience designing enterprise-scale cloud data platforms on AWS, Azure, or GCP.
• Expertise in data migration, modernization, performance tuning, and platform optimization.
• Experience with CI/CD, DevOps, Terraform/IaC, and production‑grade data platform engineering.
• Strong architecture leadership, stakeholder management, solution design, and technical mentoring skills.

Good to Have

• Experience with Lakeflow Declarative Pipelines / Delta Live Tables (DLT), Auto Loader, and Databricks Workflows.
• Knowledge of Structured Streaming, CDC, SCD, and Delta Sharing.
• Exposure to Databricks Asset Bundles, system tables, Lakehouse Federation, and serverless compute.
• Awareness of MLflow, Vector Search, RAG, and AI/ML platform integration.
• Experience In Enterprise FinOps, Chargeback/showback, And Cloud Cost Governance.

Skills: sql,tuning,databricks,migration,spark sql,apache spark,pyspark,lakehouse,unity,aws,ci/cd