Senior Databricks Architect with AI

Ciel HR; Ahmedabad District; Hybrid; INR 4,000,000 - 6,000,000; Full time

Job summary

Ciel HR in Ahmedabad invites an experienced Senior Databricks Architect with AI to lead the design and implementation of enterprise-scale data platforms on Databricks Lakehouse. The role demands strong Data Architecture, PySpark, Delta Lake, and cloud integration, with hybrid work.

Immediate joining is preferred. You will mentor teams, establish standards, and drive performance, security, and governance across data pipelines, ingestion, and modeling.

Qualifications

• We require deep expertise in Databricks Lakehouse architecture and data engineering.
• Strong background in dimensional modeling and data governance.
• Experience with PySpark, Delta Lake, and Unity Catalog.

Responsibilities

• Define end-to-end data platform architecture for enterprise scale solutions.
• Lead implementation of Databricks Lakehouse data pipelines and storage.
• Mentor teams and enforce architecture standards and best practices.
• Collaborate with stakeholders to translate requirements into scalable analytics.
• Ensure security, reliability, and performance of data solutions.

Skills

Data Architecture

Databricks Architecture

Azure Databricks

Lakehouse

Pyspark

Delta Lake

Unity Catalog

Tools

Lakeflow

ADF

ADLS Gen 2

ETL/ELT

CI/CD

Dimensional Modeling

Job description

Job Title: Senior Databricks Architect with AI

Exp:15 Years

Loc: Ahmedabad

Work Mode: Hybrid

Duration: 12 Months

Note: * We are looking for immediate joiners.

* Please apply only if you are available to join immediately and your profile matches the mandatory skills mentioned in the requirement.

Job Description

We are seeking an experienced Databricks Data Architect to lead the design architecture and implementation of enterprise scale data platforms on Databricks The ideal candidate should possess strong expertise in modern data engineering practices dimensional data modeling and Databricks Lakehouse architecture,

The candidate will act as the technical owner of the data platform providing architectural guidance establishing best practices mentoring development teams and ensuring successful delivery of scalable high performance data solutions

Skills

Mandatory Skills

• Data Architecture, Databricks Architecture, Azure Databricks, Lakehouse, Pyspark, Delta Lake, Unity Catalog

Secondary Skills

• Dimensional Modeling, Lakeflow, ADF, ADLS Gen 2, ETL/ELT, CI/CD

Key Responsibilities

• Architecture Solution Design
• Design and implement enterprise scale data platforms using the Databricks Lakehouse architecture
• Define end to end data architecture including ingestion transformation storage governance and consumption layers
• Establish architectural standards framework guidelines coding best practices and reusable design patterns
• Lead technical decisions related to scalability performance optimization reliability and security
• Conduct architecture reviews and provide recommendations to project teams

Data Engineering Databricks Implementation

• Design and build scalable ETLELT pipelines using PySpark on Databricks
• Architect and implement Bronze Silver Gold data transformation frameworks
• Design and manage Delta Lake based storage architecture including partitioning optimization and data lifecycle management
• Implement Databricks Lakeflow pipelines for data orchestration and automation
• Define data quality validation reconciliation and monitoring frameworks
• Oversee CICD and DevOps practices for data engineering projects

Data Modeling Analytics Enablement

• Design and implement enterprise data models using Dimensional Modeling techniques
• Develop Star Schemas Fact Tables Dimension Tables and conformed dimensions
• Define business metrics KPI layers and semantic models
• Architect and implement Metric Views and curated Gold datasets for reporting and analytics
• Work closely with business stakeholders to translate requirements into scalable analytical models

Technical Leadership

• Provide technical leadership and mentorship to data engineers and developers
• Perform code reviews and ensure adherence to architecture standards
• Troubleshoot complex technical issues and guide teams in solution design
• Support estimation solution proposals and technical governance activities
• Drive adoption of modern data engineering and cloudnative best practices

Required Technical Skills

Core Databricks Skills

• Databricks Lakehouse Architecture
• Databricks Workflows Lakeflow
• Delta Lake
• Unity Catalog
• Databricks SQL
• Performance Tuning Optimization
• Data Engineering
• PySpark
• Spark SQL
• ETL ELT Design
• Data Pipeline Development
• Data Quality Frameworks
• Batch and Incremental Processing
• Data Modeling
• Dimensional Data Modeling
• Star Schema Design
• Fact Dimension Modeling
• Slowly Changing Dimensions SCD
• Metric View Design
• Analytical Data Modeling
• Cloud Integration
• Azure Databricks preferred
• ADLS Gen2
• Azure Data Factory ADF
• Eventbased and Batch Ingestion Patterns
• Data Governance and Security
• Programming
• Python
• SQL
• Git
• CICD Frameworks

Desired Skills

• Data Warehouse Modernization experience
• Experience migrating legacy EDW solutions to Databricks
• Exposure to Data Mesh and Medallion Architecture
• Knowledge of Data Governance Metadata Management and Data Cataloging
• Experience supporting BI platforms such as Power BI Tableau or Looker

Leadership Expectations

The candidate should be capable of

• Owning the complete Databricks implementation lifecycle
• Defining targetstate architecture and roadmap
• Reviewing designs and code produced by team members
• Guiding developers on PySpark optimization and data modeling best practices
• Driving technical discussions with client architects and business stakeholders
• Ensuring scalability maintainability and longterm sustainability of the platform
• Providing architectural governance across multiple workstreams