Databricks Tech Lead

EXL; Bengaluru; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

EXL in Bengaluru, India seeks a Databricks Platform Engineer to design, build, and manage Databricks workspaces, clusters, and data pipelines across DEV/TEST/PROD. You will implement Unity Catalog, optimize clusters for cost, and enforce workspace security and governance.

Responsibilities include managing Delta Lake architectures, Delta Live Tables pipelines, and lakehouse patterns, plus supporting MLflow and GenAI workloads on Databricks.

Qualifications

• Experience with Databricks platform engineering.
• Proficient in PySpark/SQL data pipelines.
• Familiarity with Delta Lake architecture and Delta Live Tables.
• Knowledge of data governance and data lineage tooling.

Responsibilities

• Design, build, and maintain Databricks workspaces, clusters, and compute pools across DEV/TEST/PROD.
• Configure Unity Catalog for governance, access control, and data lineage.
• Optimize cluster configurations for cost and performance.
• Implement workspace best practices: folders, access controls, secret management.
• Manage Databricks jobs, workflows, and multi-task orchestration.
• Design Delta Lake tables with partitioning, Z-ordering, and VACUUM.
• Build Bronze/Silver/Gold lakehouse layers.
• Implement Delta Live Tables pipelines with data quality.
• Manage schema evolution, time travel, CDF.
• Integrate Delta Lake with Kafka, ADLS, S3, GCS.
• Develop PySpark/SQL pipelines; streaming from Kafka/Event Hubs/Kinesis.
• Write optimized PySpark transformations (broadcast joins, AQE, dynamic pruning).
• Create reusable transformation libraries and templates.
• Implement robust error handling, retries, and DLQ in production.
• Set up MLflow tracking servers, registries, and model lifecycles.
• Support data scientists in deploying training and inference workloads.
• Build feature pipelines with Databricks Feature Store.
• Enable GenAI workloads — fine-tuning, RAG, vector search.

Skills

Databricks

PySpark

Delta Lake

SQL

Delta Live Tables

MLflow

GenAI

Tools

Unity Catalog

Kubernetes

Job description

Job Description:

• Databricks Platform Engineering
  • Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.
  • Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.
  • Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance.
  • Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager).
  • Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.
• Delta Lake & Lakehouse Architecture
  • Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM).
  • Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.
  • Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations.
  • Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing.
  • Design data lakehouse patterns integrating Delta Lake with external systems (Kafka, ADLS, S3, GCS).
• EXL Service 10 Exchange Place, Suite 2200, Jersey City, NJ T: +1.201.748.4700 www.exlservice.com
• Data Pipeline Development (PySpark / SQL)
  • Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Lake.
  • Build structured streaming pipelines for real-time ingestion from Kafka, Event Hubs, and Kinesis into Delta tables.
  • Write optimized PySpark transformations leveraging broadcast joins, adaptive query execution (AQE), and dynamic partition pruning.
  • Create reusable transformation libraries, utility frameworks, and pipeline templates for team productivity.
  • Implement robust error handling, retry logic, and dead-letter queue patterns in production pipelines.
• MLflow & AI/ML Workloads
  • Set up and manage MLflow tracking servers, experiment registries, and model lifecycle management on Databricks.
  • Support data scientists and ML engineers in deploying model training and inference workloads on Databricks clusters and GPU instances.
  • Build feature engineering pipelines using Databricks Feature Store for reusable, versioned ML features.
  • Enable GenAI workloads — LLM fine-tuning, RAG pipeline development, and vector search (Databricks Vector Search / Mosaic AI).
  • Implement