Databricks Data & AI Platform Architect

ZoomInData; Hyderabad; Hybrid; INR 4,500,000 - 7,000,000; Full time

Job summary

ZoomInData is seeking an experienced Architect to lead the design and implementation of enterprise-scale Databricks Data & AI platforms across AWS and Azure. The role focuses on Lakehouse architectures, robust data pipelines, and governance across cloud environments.

You will drive ML/AI platform initiatives using Mosaic AI, MLflow, and GenAI solutions, enabling scalable, production-ready AI applications with strong security and observability practices.

Qualifications

• Experience architecting enterprise-scale Databricks platforms.
• Strong expertise in Delta Lake and Unity Catalog.
• Hands-on in building data pipelines with PySpark and Databricks.
• Experience with ML/AI platforms and GenAI workflows.
• Knowledge of cloud data security and governance.

Responsibilities

• Lead the architecture, design, and implementation of enterprise-scale Databricks Data & AI platforms across AWS and Azure environments.
• Define scalable Lakehouse architectures using Databricks, Delta Lake, and modern data engineering patterns for batch and real-time workloads.
• Design high-performance data pipelines and transformations using Python, SQL, PySpark, Delta Live Tables, and Databricks Workflows/Jobs.
• Architect ingestion, ETL/ELT, CDC, orchestration, and processing solutions from enterprise, cloud, and streaming sources.
• Establish data governance, security, access control, and data discovery using Unity Catalog, including cataloging, lineage, permissions, and data sharing strategies.
• Design and optimize Delta Lake architectures, including schema management, ACID transactions, partitioning, and performance tuning.
• Lead the implementation of ML and AI platforms using Databricks Mosaic AI and MLflow.
• Architect Generative AI solutions using LLMs, RAG architectures, Vector Search, embeddings, and enterprise data.
• Design and implement Vector Search and Retrieval-Augmented Generation (RAG) for enterprise knowledge discovery and AI apps.
• Establish responsible AI practices covering governance, security, evaluation, observability, and access controls for GenAI workloads.
• Integrate Databricks with AWS/Azure services, databases, APIs, and external platforms to build end-to-end data and AI solutions.
• Define standards for data modeling, pipeline development, CI/CD, DevOps, testing, monitoring, and production operations across Databricks environments.
• Drive data platform modernization and migration initiatives, transforming legacy warehouses and lakes into Databricks Lakehouse architectures.
• Optimize compute, storage, Spark workloads, SQL queries, Jobs, and Delta tables for performance, scalability, and cloud cost efficiency.
• Collaborate with data engineers, ML engineers, data scientists, cloud architects, security teams, and business stakeholders to translate requirements into technical solutions.
• Lead architecture reviews, POCs, platform evaluations, and technology strategy initiatives in Data Eng, AI/ML, GenAI.
• Troubleshoot complex data engineering, Spark, Delta Lake, ML, AI, security, and platform performance issues, providing architectural solutions and root-cause analysis.
• Provide technical leadership and mentorship to data engineers, ML engineers, and developers, establishing best practices.

Skills

Databricks

Python

SQL

PySpark

AWS

Azure

Delta Lake

Unity Catalog

Databricks Workflows/Jobs

Delta Live Tables

Mosaic AI

MLflow

Vector Search

RAG

LLMs

GenAI

Lakehouse Architecture

Data Engineering

AI/ML

Data Governance

Cloud Data Platforms

Job description

Job Responsibilities

• Lead the architecture, design, and implementation of enterprise-scale Databricks Data & AI platforms across AWS and Azure environments.
• Define and implement scalable Lakehouse architectures using Databricks, Delta Lake, and modern data engineering patterns for batch and real-time workloads.
• Design high-performance data pipelines and transformation frameworks using Python, SQL, PySpark, Delta Live Tables, and Databricks Workflows/Jobs.
• Architect robust data ingestion, ETL/ELT, CDC, orchestration, and data processing solutions integrating enterprise, cloud, and streaming data sources.
• Establish enterprise-wide data governance, security, access control, and data discovery using Unity Catalog, including cataloging, lineage, permissions, and data-sharing strategies.
• Design and optimize Delta Lake architectures, including schema management, ACID transactions, partitioning, optimization, data lifecycle management, and performance tuning.
• Lead the implementation of ML and AI platforms using Databricks Mosaic AI and MLflow, supporting model development, experiment tracking, evaluation, deployment, and lifecycle management.
• Architect Generative AI solutions using LLMs, RAG architectures, Vector Search, embeddings, and enterprise data, ensuring scalable and production-ready AI applications.
• Design and implement Vector Search and Retrieval-Augmented Generation (RAG) solutions for enterprise knowledge discovery, intelligent search, and AI-powered applications.
• Establish responsible AI practices covering model governance, data security, evaluation, observability, hallucination mitigation, and access controls for enterprise GenAI workloads.
• Integrate Databricks with AWS/Azure cloud services, enterprise applications, databases, APIs, and external data platforms to build end-to-end data and AI solutions.
• Define standards for data modeling, pipeline development, CI/CD, DevOps, testing, monitoring, observability, and production operations across Databricks environments.
• Drive data platform modernization and migration initiatives, including transformation of legacy data warehouses and data lakes into scalable Databricks Lakehouse architectures.
• Optimize compute, storage, Spark workloads, SQL queries, Jobs, and Delta tables to improve performance, scalability, reliability, and cloud cost efficiency.
• Collaborate with data engineers, ML engineers, data scientists, cloud architects, security teams, and business stakeholders to translate business requirements into technical solutions.
• Lead architecture reviews, technical POCs, platform evaluations, and technology strategy initiatives related to Data Engineering, AI/ML, and GenAI.
• Troubleshoot complex data engineering, Spark, Delta Lake, ML, AI, security, and platform performance issues, providing architectural solutions and root-cause analysis.
• Provide technical leadership and mentorship to data engineers, ML engineers, and developers, establishing best practices and ensuring delivery against enterprise architecture standards.

Key Technical Skills

Databricks | Python | SQL | PySpark | AWS | Azure | Delta Lake | Unity Catalog | Databricks Workflows/Jobs | Delta Live Tables | Mosaic AI | MLflow | Vector Search | RAG | LLMs | GenAI | Lakehouse Architecture | Data Engineering | AI/ML | Data Governance | Cloud Data Platforms