Databricks Senior Architect

Vrinda Global; Bengaluru; Vor Ort; INR 3.000.000 - 6.000.000; Vollzeit

Zusammenfassung

Vrinda Global is seeking a Senior Databricks Architect to lead enterprise-scale Lakehouse architecture and data platform modernization in Bengaluru. You will define target-state architecture, governance, and migration strategies while driving performance, cost efficiency, and AI-ready data foundations.

You will collaborate with stakeholders to translate business requirements into scalable, secure, and reliable data solutions, overseeing analytics pipelines, data contracts, and platform

Qualifikationen

• Extensive hands-on experience with Databricks Lakehouse Platform and enterprise data architecture.
• Expertise in Apache Spark, PySpark, Spark SQL, Delta Lake, and Structured Streaming.
• Strong understanding of Unity Catalog, Medallion Architecture, DLT/Lakeflow, Auto Loader, and Databricks Workflows.
• Experience with data warehouse modernization, Hadoop migration, and ETL transformation.
• Strong knowledge of data governance, security, access controls, and data-sharing frameworks.
• Experience in CI/CD, Git, Infrastructure as Code, DataOps, and production deployment practices.
• Expertise in Spark performance tuning, workload optimization, and cloud cost management.
• Working knowledge of MLflow, Vector Search, Model Serving, and AI-ready data architectures.

Aufgaben

• Define enterprise-scale Databricks Lakehouse architecture, workspace strategy, environment setup, and Unity Catalog architecture.
• Establish standards for Delta Lake, DLT, Workflows, Auto Loader, and Databricks Asset Bundles.
• Design reusable architecture blueprints, reference architectures, and technical implementation standards.
• Define platform architecture principles, non-functional requirements, and engineering guardrails.
• Establish engineering standards for PySpark, Spark SQL, Delta Lake, Structured Streaming, CDC, SCD, and incremental data processing.
• Drive optimization of Spark workloads through execution-plan analysis, AQE, partitioning, caching, Photon, and shuffle/skew management.
• Design scalable, reliable, and recoverable batch and streaming data pipelines.
• Define standards for data contracts, schema evolution, and pipeline orchestration.
• Lead modernization initiatives involving legacy data warehouses, Hadoop ecosystems, and traditional ETL platforms.
• Define migration strategies, roadmaps, conversion patterns, and implementation plans.
• Drive data validation, reconciliation, parallel runs, cutover, rollback, and decommissioning strategies.
• Develop reusable migration frameworks and accelerators to minimize delivery risks and business disruption.
• Design and implement Unity Catalog governance frameworks, including catalogs, schemas, external locations, and service principals.
• Define RBAC/ABAC, data masking, row-level and column-level security, and access management standards.
• Establish data lineage, quality, privacy, retention, and auditability frameworks.
• Drive governed data-sharing practices using Delta Sharing and Lakehouse Federation.
• Define Git-based CI/CD pipelines, automated testing, infrastructure-as-code, and deployment strategies for Databricks.
• Establish platform monitoring, observability, alerting, SLIs/SLOs, and production-readiness standards.
• Drive platform reliability, operational excellence, and incident management practices.
• Build reusable frameworks, templates, and self-service engineering capabilities.
• Define compute and cluster policies, autoscaling strategies, workload sizing, and serverless adoption guidelines.
• Establish cost monitoring, tagging, chargeback/showback, and FinOps practices.
• Identify opportunities for performance and cost optimization across Databricks workloads.
• Balance platform performance, scalability, reliability, and infrastructure costs.
• Design governed and high-quality data foundations to support Generative AI, RAG, and enterprise AI use cases.
• Ensure data quality, lineage, security, and retrieval readiness for AI-driven applications.
• Demonstrate working knowledge of Databricks Vector Search, MLflow, Model Serving, model lifecycle management, and AI governance.
• Evaluate emerging Databricks AI capabilities and recommend suitable enterprise adoption strategies.
• Lead technical discovery sessions, architecture workshops, and solution-design discussions.
• Present architecture options, technical trade-offs, risks, roadmaps, and estimates to business and technology stakeholders.
• Mentor engineering teams and technical leads, review architecture designs, and ensure implementation quality.
• Support solution proposals, client presentations, and customer-facing architecture discussions.

Kenntnisse

Databricks Lakehouse Platform

Apache Spark

PySpark

Spark SQL

Delta Lake

Unity Catalog

DLT / Lakehouse tooling

Data governance

Cloud cost management

CI/CD

Git

Infrastructure as Code

MLOps

Tools

Git

Jobbeschreibung

Locations: Bangalore, Pune, Hyderabad, Chennai, and Noida

Role Overview

We are looking for an experienced Senior Databricks Architect to lead enterprise-scale Lakehouse architecture, data platform modernization, governance, and engineering transformation initiatives.

The ideal candidate will have strong expertise in Databricks, Apache Spark, PySpark, Delta Lake, Unity Catalog, and modern data engineering architectures. The role involves defining target-state architecture, establishing engineering standards, driving migration strategies, optimizing platform performance and costs, and building AI-ready data foundations.

The candidate will work closely with business stakeholders, technology leaders, and engineering teams to translate business requirements into scalable, secure, reliable, and cost-efficient data platform solutions.

Key Responsibilities

1. Lakehouse & Platform Architecture

• Define enterprise-scale Databricks Lakehouse architecture, workspace strategy, environment setup, and Unity Catalog architecture.
• Establish standards for Medallion Architecture, Delta Lake, Lakeflow Declarative Pipelines (DLT), Workflows, Auto Loader, and Databricks Asset Bundles.
• Design reusable architecture blueprints, reference architectures, and technical implementation standards.
• Define platform architecture principles, non-functional requirements, and engineering guardrails.
• Establish engineering standards for PySpark, Spark SQL, Delta Lake, Structured Streaming, CDC, SCD, and incremental data processing.
• Drive optimization of Spark workloads through execution-plan analysis, AQE, partitioning, caching, Photon, and shuffle/skew management.
• Design scalable, reliable, and recoverable batch and streaming data pipelines.
• Define standards for data contracts, schema evolution, and pipeline orchestration.

3. Data Migration & Modernization

• Lead modernization initiatives involving legacy data warehouses, Hadoop ecosystems, and traditional ETL platforms.
• Define migration strategies, roadmaps, conversion patterns, and implementation plans.
• Drive data validation, reconciliation, parallel runs, cutover, rollback, and decommissioning strategies.
• Develop reusable migration frameworks and accelerators to minimize delivery risks and business disruption.

4. Data Governance, Security & Compliance

• Design and implement Unity Catalog governance frameworks, including catalogs, schemas, external locations, and service principals.
• Define RBAC/ABAC, data masking, row-level and column-level security, and access management standards.
• Establish data lineage, quality, privacy, retention, and auditability frameworks.
• Drive governed data-sharing practices using Delta Sharing and Lakehouse Federation.

5. Platform Engineering, DevOps & Reliability

• Define Git-based CI/CD pipelines, automated testing, infrastructure-as-code, and deployment strategies for Databricks.
• Establish platform monitoring, observability, alerting, SLIs/SLOs, and production-readiness standards.
• Drive platform reliability, operational excellence, and incident management practices.
• Build reusable frameworks, templates, and self-service engineering capabilities.

6. Performance Optimization & FinOps

• Define compute and cluster policies, autoscaling strategies, workload sizing, and serverless adoption guidelines.
• Establish cost monitoring, tagging, chargeback/showback, and FinOps practices.
• Identify opportunities for performance and cost optimization across Databricks workloads.
• Balance platform performance, scalability, reliability, and infrastructure costs.

7. AI-Ready Data Foundations

• Design governed and high-quality data foundations to support Generative AI, RAG, and enterprise AI use cases.
• Ensure data quality, lineage, security, and retrieval readiness for AI-driven applications.
• Demonstrate working knowledge of Databricks Vector Search, MLflow, Model Serving, model lifecycle management, and AI governance.
• Evaluate emerging Databricks AI capabilities and recommend suitable enterprise adoption strategies.

8. Consulting & Stakeholder Leadership

• Lead technical discovery sessions, architecture workshops, and solution-design discussions.
• Present architecture options, technical trade-offs, risks, roadmaps, and estimates to business and technology stakeholders.
• Mentor engineering teams and technical leads, review architecture designs, and ensure implementation quality.
• Support solution proposals, client presentations, and customer-facing architecture discussions.

Required Technical Skills

• Strong hands-on experience with Databricks Lakehouse Platform and enterprise data architecture.
• Expertise in Apache Spark, PySpark, Spark SQL, Delta Lake, and Structured Streaming.
• Strong understanding of Unity Catalog, Medallion Architecture, DLT/Lakeflow, Auto Loader, and Databricks Workflows.
• Experience with data warehouse modernization, Hadoop migration, and ETL transformation.
• Strong knowledge of data governance, security, access controls, and data-sharing frameworks.
• Experience in CI/CD, Git, Infrastructure as Code, DataOps, and production deployment practices.
• Expertise in Spark performance tuning, workload optimization, and cloud cost management.
• Working knowledge of MLflow, Vector Search, Model Serving, and AI-ready data architectures.

Preferred Experience

• Experience defining enterprise-wide Databricks architecture and platform standards.
• Experience leading large-scale data migration and modernization programs.
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Experience in architecture consulting, stakeholder management, and technical team leadership.
• Ability to independently drive architecture decisions and guide engineering teams through implementation.