Senior Databricks Architect

Zorba AI; Chennai; Presencial; INR 4.000.000 - 7.000.000; Jornada completa

Descripción de la vacante

Zorba AI in Chennai, India, is seeking an experienced Senior Databricks Architect to lead enterprise-scale Lakehouse architecture, platform modernization, and data governance. You will define architecture strategies, standards, and roadmaps for Databricks, Unity Catalog, and Delta Lake.

The role requires hands-on expertise in Databricks, Spark, PySpark, Delta Lake, and cloud platforms (AWS/Azure/GCP), plus strong leadership to mentor teams and drive cost optimization, CI/CD, and DevOps practices.

Formación

• 10–15 years of IT experience with data engineering and architecture.
• Strong hands-on Databricks Lakehouse, Spark, PySpark, and Delta Lake.
• Expertise in Unity Catalog, data governance, security, and lineage.
• Experience designing cloud data platforms on AWS/Azure/GCP.
• Proven ability to drive modernization, performance tuning, and cost optimization.
• Leadership in architecture, mentoring, and stakeholder collaboration.

Responsabilidades

• Design and lead enterprise Databricks Lakehouse architecture and Medallion patterns.
• Define standards for Databricks, PySpark, Spark SQL, Delta Lake, and DLT.
• Lead modernization and migration from legacy systems to Databricks.
• Drive Spark performance optimization, AQE, partitioning, caching, and Photon.
• Establish data governance, security, and Unity Catalog frameworks.
• Set CI/CD, DevOps, Terraform/IaC, and automated testing standards.
• Define platform reliability, monitoring, and disaster recovery plans.
• Lead architecture workshops and mentor engineering teams.

Conocimientos

Databricks Lakehouse

Apache Spark

PySpark

Delta Lake

Unity Catalog

Cloud Platforms (AWS/Azure/GCP)

CI/CD

Terraform/IaC

DevOps

Architecture Leadership

Herramientas

Delta Live Tables

Databricks Workflows

Auto Loader

Descripción del empleo

We are looking for an experienced Senior Databricks Architect with 10–15 years of IT experience to lead enterprise-scale Lakehouse architecture, platform modernization, data governance, and migration initiatives. The candidate should have strong hands‑on expertise in Databricks, Apache Spark, PySpark, Delta Lake, and Unity Catalog, along with the ability to define architecture strategies, engineering standards, performance optimization, and cloud data platform roadmaps.

Key Responsibilities

• Design and lead enterprise-scale Databricks Lakehouse architecture, including workspace strategy, Unity Catalog, storage, compute, and Medallion architecture.
• Define architecture standards and best practices for Databricks, PySpark, Spark SQL, Delta Lake, DLT/Lakeflow, Workflows, and Auto Loader.
• Lead large-scale data platform modernization and migration from legacy data warehouses, Hadoop, and ETL platforms to Databricks.
• Drive Spark performance optimization, including query tuning, AQE, partitioning, caching, shuffle optimization, and Photon.
• Design enterprise data governance, security, access controls, lineage, data quality, and Delta Sharing frameworks using Unity Catalog.
• Establish CI/CD, DevOps, Infrastructure as Code (Terraform), automated testing, and deployment standards for Databricks platforms.
• Define platform reliability, monitoring, observability, disaster recovery, and production support standards.
• Drive Databricks cost optimization through cluster policies, autoscaling, compute sizing, serverless adoption, and FinOps practices.
• Build AI‑ready data foundations and demonstrate awareness of RAG, Vector Search, MLflow, Model Serving, and AI governance.
• Lead architecture workshops, engage with business and technical stakeholders, define technology roadmaps, and mentor engineering teams.

Mandatory Skills

• 10–15 years of overall IT experience with significant experience in Data Engineering and Architecture.
• Strong expertise in Databricks Lakehouse Architecture and Apache Spark.
• Hands‑on experience with PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
• Strong knowledge of Unity Catalog, Data Governance, Security, and Data Lineage.
• Experience designing enterprise-scale cloud data platforms on AWS, Azure, or GCP.
• Expertise in data migration, modernization, performance tuning, and platform optimization.
• Experience with CI/CD, DevOps, Terraform/IaC, and production‑grade data platform engineering.
• Strong architecture leadership, stakeholder management, solution design, and technical mentoring skills.

Good to Have

• Experience with Lakeflow Declarative Pipelines / Delta Live Tables (DLT), Auto Loader, and Databricks Workflows.
• Knowledge of Structured Streaming, CDC, SCD, and Delta Sharing.
• Exposure to Databricks Asset Bundles, system tables, Lakehouse Federation, and serverless compute.
• Awareness of MLflow, Vector Search, RAG, and AI/ML platform integration.
• Experience In Enterprise FinOps, Chargeback/showback, And Cloud Cost Governance.

Skills: sql,tuning,databricks,migration,spark sql,apache spark,pyspark,lakehouse,unity,aws,ci/cd