Databricks Architect

Intellics Global Services; Chennai District; On-site; INR 1,100,000 - 1,900,000; Full time

Job summary

Intellics Global Services is seeking a senior Databricks Architect to design, modernize, and optimize the Lakehouse architecture. You will refactor legacy pipelines using PySpark, Databricks SQL, or Delta Live Tables and drive performance improvements across batch and streaming workloads.

The role requires deep knowledge of Delta Lake, Unity Catalog, and advanced SQL with hands-on experience in scalable data pipelines and MLflow integration.

Qualifications

• Refactor data pipelines and SQL/scripts to PySpark/Databricks
• Knowledge of Delta Lake and Delta Live Tables
• Familiarity with Unity Catalog and lakehouse architecture
• Ability to design batch and streaming ingestion patterns

Responsibilities

• Refactor legacy Synapse T-SQL stored procedures, scripts, and views into native PySpark, Databricks SQL, or Delta Live Tables (DLT)
• Transition legacy Synapse distribution schemas to Delta Lake Liquid Clustering (CLUSTER BY)
• Design efficient batch and streaming ingestion patterns using Auto Loader and Delta Lake Lakehouse mechanisms
• Conduct performance tuning (OPTIMIZE, Z-ORDER, compute cluster sizing, Serverless warehouse setup)

Skills

Databricks Lakehouse

Delta Live Tables

Unity Catalog

Partitioning Strategies

Python

PySpark

Advanced SQL

Scala (optional)

Tools

Databricks Runtime

Delta Lake

Delta Live Tables

Auto Loader

DB SQL Serverless

MLflow

dbx

Git

Job description

Role Overview

Serves as the core technical authority for designing, modernizing, and optimizing the Databricks Lakehouse architecture, code refactoring, and data execution pipelines.

Roles & Responsibilities

• Refactor legacy Synapse T-SQL stored procedures, scripts, and views into native PySpark, Databricks SQL, or Delta Live Tables (DLT)
• Transition legacy Synapse distribution schemas (HASH, ROUND_ROBIN, REPLICATE) to Delta Lake Liquid Clustering (CLUSTER BY) for optimal query performance and maintenance
• Design efficient batch and streaming ingestion patterns using Auto Loader and Delta Lake Lakehouse mechanisms
• Conduct performance tuning (OPTIMIZE, Z-ORDER, compute cluster sizing, Serverless warehouse setup)

Key Skills Required

• Designing end-to-end Target State Architectures Blueprint incorporating Databricks Lakehouse components (Unity Catalog, Delta Live Tables)
• Databricks: Databricks Runtime, Delta Lake, Delta Live Tables (DLT)
• Articulating design decisions, evaluating build vs. buy, and balancing performance with licensing costs
• Performance & Storage: Liquid Clustering, Partitioning Strategies, Auto Loader, DB SQL Serverless
• Languages: Python, PySpark, Advanced SQL, Scala (optional)

CI/CD & Tools: Databricks Asset Bundles (DABs), dbx, Git integration, MLflow