Databricks Architect

SPG Consulting; Bengaluru Urban; On-site; INR 4,500,000 - 6,500,000; Full time

Job summary

SPG Consulting in Bangalore is seeking an experienced Databricks Architect to design scalable, secure data platforms using Databricks and Apache Spark. The role demands deep expertise in Lakehouse architectures, cloud data services, and enterprise-scale implementations.

You will lead data engineering teams, define governance, implement Unity Catalog, and drive CI/CD for notebooks, jobs, and pipelines while optimizing costs and performance.

Qualifications

• Bachelor's or Master's degree in CS/IT/Engineering/Data Science.
• Strong expertise in Databricks Lakehouse architectures.
• Experience leading enterprise data platforms and modernization programs.

Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures.
• Define data architecture strategies, standards and governance.
• Develop ingestion, transformation, processing, and analytics pipelines.
• Provide technical leadership to Data Engineering and BI teams.
• Review existing data platforms and recommend modernization strategies.

Skills

Databricks

Apache Spark

PySpark

Scala

Delta Lake

Unity Catalog

Databricks SQL

Azure AWS GCP

ADLS

ADF

Kafka

Terraform

Git

CI/CD

MLflow

Education

Bachelor's or Master's in Computer Science, Information Technology, Engineering, Data Science

Job description

Bangalore North, India | Posted on 08/19/2026

We are looking for an experienced Databricks Architect to design and implement scalable, secure, and high-performance data platforms using Databricks and Apache Spark. The ideal candidate should have strong expertise in data architecture, cloud platforms, data engineering, Lakehouse architecture, and enterprise-scale Databricks implementations.

Key Responsibilities

Design and implement enterprise-grade Databricks Lakehouse architectures.

Define data architecture strategies, standards, governance, and best practices.

Design scalable data ingestion, transformation, processing, and analytics pipelines.

Develop architecture solutions using Databricks, Apache Spark, Delta Lake, and cloud-native services.

Define data lake and lakehouse structures using Bronze, Silver, and Gold layers.

Design batch and real-time data processing solutions.

Provide technical leadership to Data Engineering and BI teams.

Review existing data platforms and recommend modernization strategies.

Design high-performance and cost-optimized Databricks solutions.

Establish security, access control, encryption, and data governance standards.

Design and implement Unity Catalog for centralized data governance.

Define data lineage, discovery, auditing, and access-management strategies.

Design CI/CD and DevOps processes for Databricks notebooks, jobs, workflows, and code.

Integrate Databricks with enterprise data sources, APIs, warehouses, and cloud storage.

Lead migration of legacy data platforms to Databricks where applicable.

Conduct architecture reviews, proof-of-concepts, and technology evaluations.

Collaborate with Data Engineers, Data Scientists, BI Developers, Cloud Architects, and business stakeholders.

Provide technical guidance, mentoring, and architectural documentation.

Required Skills

Databricks & Spark

Strong hands-on experience with Databricks.

Expert-level knowledge of Apache Spark.

Strong experience with PySpark and/or Scala.

Experience with Delta Lake and Delta tables.

Strong knowledge of Databricks Workflows, Jobs, Clusters, Notebooks, and SQL Warehouses.

Experience designing and implementing Lakehouse architecture.

Knowledge of Spark performance tuning and optimization.

Strong understanding of:

Data Lake / Data Warehouse / Lakehouse

Medallion Architecture

Data Modeling

ETL/ELT

Batch and Streaming

Data Governance

Data Quality

Metadata Management

Experience designing enterprise data platforms and integration architectures.

Cloud

Strong experience with at least one major cloud platform:

Amazon Web Services (AWS)

Preferred Azure technologies include:

Unity Catalog & Governance

Strong knowledge of Databricks Unity Catalog.

Design and implement catalogs, schemas, external locations, and storage credentials.

Implement role-based access control and data security.

Establish data lineage and auditing.

Define enterprise data governance and compliance practices.

DevOps & CI/CD

Experience implementing CI/CD for Databricks solutions.

Knowledge of Git/GitHub, Azure DevOps, GitHub Actions, or Jenkins.

Experience with Infrastructure as Code such as Terraform.

Familiarity with automated testing, deployment, and release management.

Performance & Cost Optimization

Optimize Spark workloads, SQL queries, clusters, and Delta tables.

Experience with partitioning, caching, file optimization, and indexing strategies.

Knowledge of Delta Lake OPTIMIZE, VACUUM, Z-Ordering, and liquid clustering.

Implement appropriate cluster sizing and autoscaling strategies.

Monitor and optimize Databricks platform costs.

Good to Have

Experience with Databricks SQL and BI integration.

Knowledge of MLflow and Machine Learning workloads.

Experience with streaming technologies such as Kafka or Azure Event Hubs.

Knowledge of Terraform and Infrastructure as Code.

Experience with Generative AI, RAG, or Vector Search.

Familiarity with Microsoft Fabric or Snowflake.

Experience with large-scale cloud migration projects.

Knowledge of enterprise security and regulatory requirements.

Key Technologies

Databricks | Apache Spark | PySpark | Scala | Delta Lake | Unity Catalog | Databricks SQL | Azure/AWS/GCP | ADLS | ADF | Kafka | Terraform | Git | CI/CD | MLflow

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.

Preferred Candidate Profile

The ideal candidate should have strong experience designing enterprise Databricks Lakehouse platforms and leading complex data engineering initiatives. The candidate should combine hands-on Databricks expertise with strong knowledge of cloud architecture, Spark, Delta Lake, Unity Catalog, security, governance, performance optimization, and DevOps.

Experience leading large-scale Databricks migration and modernization programs will be highly valued.