Databricks Tech Lead

EXL; Chennai; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

EXL in Chennai, India seeks a Databricks Engineer to design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. You will deliver pipelines for AML, KYC, CRA, sanctions screening, transaction monitoring, fraud detection, and regulatory reporting.

The role emphasizes Delta Lake architecture, Delta Live Tables pipelines, and strong data governance. 8+ years in data/soft engineering and 3+ years on Databricks production environments are preferred.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.
• 8+ years of total experience in data engineering or software engineering.
• 3+ years of dedicated hands-on experience with the Databricks platform in production environments.
• Strong background in big data engineering, cloud data platforms, and distributed computing.

Responsibilities

• Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.
• Configure and manage Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.
• Optimize cluster configurations for cost and performance.
• Implement workspace-level best practices: folder structures, access controls, secret management.
• Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.
• Design and implement Delta Lake tables with partitioning and optimization.
• Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.
• Implement Delta Live Tables (DLT) pipelines with data quality expectations.
• Manage schema evolution, versioning, time travel, and Change Data Feed.
• Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS).

Skills

Databricks platform expertise

Big data engineering

Cloud data platforms

Distributed computing

Data security & governance

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field

Tools

Databricks

Unity Catalog

Delta Live Tables

Delta Lake

Databricks SQL Warehouses

Job description

Job Description:

We are looking for a skilled and passionate Databricks Engineer to design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. The successful candidate will be responsible for creating Databricks pipeline delivering Financial Crime platforms covering Anti-Money Laundering (AML), Know Your Customer (KYC), Customer Risk Assessment (CRA), Sanctions Screening, Transaction Monitoring, Fraud Detection, and Regulatory Reporting

Responsibilities

Databricks Platform Engineering

• Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.

• Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.

• Optimize cluster configurations - instance types, auto-scaling policies, spot/preemptible nodes - for cost and performance.

• Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager).

• Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.

Delta Lake & Lakehouse Architecture

• Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM).

• Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.

• Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations.

• Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing.

• Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS).

Qualifications

Education

• Bachelors or Masters degree in Computer Science, Information Technology, Data Engineering, or related field.

Experience

• 8+ years of total experience in data engineering or software engineering.

• 3+ years of dedicated hands‑on experience with the Databricks platform in production environments.

• Strong background in big data engineering, cloud data platforms, and distributed computing.

Databricks Platform

• Deep expertise in Databricks Workspaces, Clusters, Jobs, Workflows, and Repos.

• Proficiency with Unity Catalog - metastore setup, catalog/schema/table management, access controls, and data lineage.

• Hands‑on experience with Delta Live Tables (DLT) - pipeline development, expectations, and monitoring.

• Strong command of Delta Lake internals - transaction log, ACID guarantees, file layout, and optimization techniques.

• Experience with Databricks SQL Warehouses, SQL Analytics, and dashboard creation.

• Knowledge of Databricks Photon engine, serverless compute, and cost optimization strategies.

Requirements: