Databricks Architect

Maneva Consulting; Chennai District; Hybrid; INR 2,500,000 - 4,500,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

Maneva Consulting in Chennai (Hybrid) seeks a Databricks Architect with 15-20 years of experience to lead the transformation of legacy systems to modern Databricks Lakehouse solutions. The role demands designing scalable data pipelines, optimizing performance, and balancing cost considerations.

The candidate will work on batch and streaming ingestion, Delta Lake—DLT, and Unity Catalog, while collaborating with cross-functional teams to deliver a robust data platform with strong governance.

Qualifications

• Senior role requiring 15+ years in data engineering/architecture.
• Experience with Databricks Lakehouse; exposure to Unity Catalog and DLT.
• Strong design skills for end-to-end architectures balancing performance and cost.
• Proficiency in Python/SQL; knowledge of Scala is a plus.

Responsibilities

• Refactor and migrate legacy Data Platform components to Databricks.
• Design batch and streaming ingestion using Autoloader and Delta Lake Lakehouse.
• Tune performance with OPTIMIZE, Z-ORDER and appropriate cluster sizing.
• Collaborate on cost-optimized architectures balancing licensing impacts.

Skills

Target Architecture

Databricks

Cost Optimization

Performance Storage

Python

SQL

Scala

CI/CD Tools

Tools

Databricks Runtime

Delta Lake

Delta Live Tables

Autoloader

Git Integration

MLflow

dbx

DABs

Job description

Greetings from Maneva!

Job Description

Job Title - Databricks Architect

Experience - 15 - 20 Years

Location - Chennai (Hybrid)

Notice - Immediate to 30 days

Requirements:-

• Refactor legacy Synapse T-SQL stored procedures, scripts, and views into native PySpark, Databricks SQL, or Delta Live Tables (DLT).
• Transition legacy Synapse distribution schemas (HASH, ROUND_ROBIN, REPLICATE) to Delta Lake Liquid Clustering (CLUSTER BY) for optimal query performance and maintenance.
• Design efficient batch and streaming ingestion patterns using Autoloader and Delta Lake Lakehouse mechanisms.
• Conduct performance tuning using OPTIMIZE, Z-ORDER, compute cluster sizing, and Serverless Warehouse setup.

Key Skills

• Target State Architecture: Designing end-to-end Target State Architecture Blueprints incorporating Databricks Lakehouse components, Unity Catalog, and Delta Live Tables (DLT).
• Databricks: Databricks Runtime, Delta Lake, Delta Live Tables (DLT).
• Architecture & Cost Optimization: Articulating design decisions, evaluating build vs. buy, and balancing performance with licensing costs.
• Performance & Storage: Liquid Clustering, Partitioning Strategies, Auto Loader, Databricks SQL Serverless.
• Languages: Python, PySpark, Advanced SQL, Scala (Optional).
• CI/CD & Tools: Databricks Asset Bundles (DABs), dbx, Git Integration, MLflow.