Databricks Engineer

EXL; Pune District; On-site; INR 3,500,000 - 5,500,000; Full time

Job summary

EXL is seeking a seasoned Databricks Engineer to design and optimize enterprise-scale data lakehouse solutions on the Databricks platform. You will build and maintain pipelines covering AML/KYC/CRA, Sanctions, Transaction Monitoring, Fraud, and Regulatory Reporting.

You will configure Unity Catalog, Delta Lake, DLT pipelines, and Medallion Architecture, while enabling scalable ML workflows with MLflow and feature stores. Strong cloud experience and performance tuning are essential.

Qualifications

• Bachelor’s or master’s degree in CS/IT/related field or equivalent practical experience.
• 6+ years total experience in data engineering or software engineering.
• 3+ years hands-on Databricks platform experience in production environments.
• Strong background in big data, cloud data platforms, and distributed computing.
• Deep expertise in Databricks Workspaces, Clusters, Jobs, Workflows, and Repos.
• Proficiency with Unity Catalog — metastore setup, catalog/schema/table management, access controls, and data lineage.
• Hands-on experience with Delta Live Tables (DLT) — pipeline development, expectations, and monitoring.
• Strong SQL knowledge with advanced analytics (window functions, joins).
• Experience with Spark performance tuning — AQE, caching, partitioning.

Responsibilities

• Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production.
• Configure and manage Unity Catalog for data governance, access controls, metadata management, and data lineage.
• Optimize cluster configurations for performance and cost efficiency.
• Implement best practices for folders, secrets, and governance in Databricks.
• Create, schedule, and manage Databricks Jobs, Workflows, and multi-task orchestration.
• Design Delta Lake tables with partitioning, Z-Ordering, OPTIMIZE, VACUUM, and compaction.
• Build Medallion Architecture (Bronze/Silver/Gold) for scalable data lakehouse solutions.
• Develop Delta Live Tables pipelines with data quality expectations for ETL/ELT.
• Manage schema evolution, versioning, Time Travel, and Change Data Feed for incremental processing.
• Design lakehouse architectures integrating Delta Lake with ADLS, Kafka, Event Hubs, Kinesis.
• Develop batch and real-time pipelines using PySpark, Spark SQL, Structured Streaming.
• Build streaming ingestion pipelines from Kafka, Event Hubs, and others into Delta tables.
• Optimize PySpark apps using broadcast joins, AQE, dynamic pruning, caching, and Photon engine.
• Create reusable transformation frameworks and templates for productivity.
• Implement robust error handling, retries, logging, monitoring, DLQ patterns for production.
• Set up MLflow experiments, model registry, and lifecycle management.
• Support ML workloads with scalable training, inference, and GPU compute environments.
• Develop feature pipelines with Databricks Feature Store.

Skills

Databricks

PySpark

Delta Lake

Delta Live Tables

Unity Catalog

SQL

Python

Cloud platforms

Data governance

MLflow

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Azure

AWS

GCP

ML tooling

Job description

We are looking for a skilled and passionate Databricks Engineer to design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. The successful candidate will be responsible for creating Databricks pipeline delivering Financial Crime platforms covering Anti-Money Laundering (AML), Know Your Customer (KYC), Customer Risk Assessment (CRA), Sanctions Screening, Transaction Monitoring, Fraud Detection, and Regulatory Reporting

Responsibilities for Internal Candidates

• Design, build, and maintain Databricks workspaces, clusters, and compute pools across development, testing, and production environments.
• Configure and manage Unity Catalog for data governance, fine-grained access control, permissions, metadata management, and data lineage.
• Optimize Databricks cluster configurations, including instance types, auto-scaling, spot/preemptible nodes, and compute pools to improve performance and reduce costs.
• Implement workspace best practices, including folder structures, access controls, secret management using Databricks Secrets, Azure Key Vault, or AWS Secrets Manager.
• Create, schedule, and manage Databricks Jobs, Workflows, and multi-task job orchestration with dependency management.
• Design and implement Delta Lake tables using partitioning, Z-Ordering, OPTIMIZE, VACUUM, and file compaction techniques.
• Build and maintain Medallion Architecture (Bronze, Silver, and Gold layers) for scalable and governed data lakehouse solutions.
• Develop Delta Live Tables (DLT) pipelines with built-in data quality expectations for reliable ETL/ELT processing.
• Manage schema evolution, table versioning, Time Travel, and Change Data Feed (CDF) to support incremental data processing.
• Design and implement lakehouse architectures integrating Delta Lake with cloud storage and external systems such as Azure Data Lake Storage (ADLS), Kafka, Event Hubs, and Kinesis.
• Develop scalable batch and real-time data pipelines using PySpark, Spark SQL, Structured Streaming, and Delta Lake.
• Build streaming ingestion pipelines from Kafka, Azure Event Hubs, and other streaming platforms into Delta tables.
• Optimize PySpark applications using broadcast joins, Adaptive Query Execution (AQE), dynamic partition pruning, caching, and Photon Engine.
• Develop reusable transformation frameworks, utility libraries, and pipeline templates to improve engineering productivity and standardization.
• Implement robust error handling, retry mechanisms, logging, monitoring, and dead-letter queue (DLQ) patterns for production-grade pipelines.
• Set up and manage MLflow experiment tracking, model registry, and model lifecycle management.
• Support machine learning workloads by enabling scalable model training, inference, and GPU-based compute environments.
• Develop feature engineering pipelines using Databricks Feature Store to create reusable and versioned machine learning features.
• Enable Generative AI solutions, including Retrieval-Augmented Generation (RAG), vector search, LLM fine-tuning, and Mosaic AI capabilities.
• Implement MLOps best practices, including model versioning, model deployment, A/B testing, and Databricks Model Serving.
• Integrate Databricks with Azure Data Lake Storage (ADLS) and other cloud-native services.
• Develop and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or GitLab CI for Databricks notebooks, jobs, and workflows.
• Automate Databricks infrastructure deployment using Databricks Asset Bundles (DABs), Terraform, and Infrastructure-as-Code (IaC) practices.
• Build and manage data ingestion frameworks using Auto Loader, COPY INTO, and third-party integration tools such as Fivetran, dbt, and Airbyte.
• Monitor pipeline execution, cluster utilization, system performance, and cloud costs using Databricks system tables and cloud monitoring tools.
• Implement row-level security, column-level masking, dynamic views, and governance policies using Unity Catalog.
• Enforce data quality through Delta Live Tables expectations and Great Expectations frameworks.
• Perform query optimization, execution plan analysis, caching strategies, and performance tuning to improve workload efficiency.
• Maintain enterprise data cataloging, metadata management, and end-to-end data lineage.
• Prepare technical documentation, architecture diagrams, operational runbooks, and standard operating procedures for Databricks platform and data engineering solutions.

Qualifications for Internal Candidates

• Bachelor’s or master’s degree in computer science, Information Technology, Data Engineering, or related field.
• 6+ years of total experience in data engineering or software engineering.
• 3+ years of dedicated hands-on experience with the Databricks platform in production environments.
• Strong background in big data engineering, cloud data platforms, and distributed computing.
• Deep expertise in Databricks Workspaces, Clusters, Jobs, Workflows, and Repos.
• Proficiency with Unity Catalog — metastore setup, catalog/schema/table management, access controls, and data lineage.
• Hands-on experience with Delta Live Tables (DLT) — pipeline development, expectations, and monitoring.
• Strong command of Delta Lake internals — transaction log, ACID guarantees, file layout, and optimization techniques.
• Experience with Databricks SQL Warehouses, SQL Analytics, and dashboard creation.
• Knowledge of Databricks Photon engine, serverless compute, and cost optimization strategies.
• 4+ years of PySpark development — Dataframe, Datasets, Spark SQL, RDD operations.
• Expert-level SQL — window functions, lateral joins, CTEs, recursive queries, and analytical functions.
• Experience with Spark performance tuning — AQE, query plans (EXPLAIN), partitioning, and caching.
• Proficiency with Python for pipeline development, utilities, and automation.
• Hands-on experience with at least one: Azure (ADLS Gen2, ADF, Azure Databricks), AWS (S3, EMR, Glue, AWS Databricks), or GCP (GCS, BigQuery, Dataproc).
• Experience with cloud networking for Databricks: VNet/VPC injection, private endpoints, and firewall configurations.
• Familiarity with IAM roles, managed identities, and service principal authentication for Databricks.

MLflow & ML Engineering (Nice to Have):

• Working knowledge of MLflow — experiment tracking, model registry, and deployment.
• Experience supporting ML pipelines on Databricks for training, evaluation, and serving.