Senior Databricks Engineer

EXL; United States; On-site; USD 130,000 - 195,000; Full time

Benefits offered by this job

Mentoring program

Growth opportunities

Job summary

EXL is seeking a Senior Databricks Engineer to independently design, build and own production-grade data pipelines on the Databricks Lakehouse. You will deliver Bronze/Silver/Gold data products for analytics and ML, using PySpark, Spark SQL, and Delta Live Tables.

You will implement complex transformations, tune performance, and ensure observability and governance with Unity Catalog and Azure data services. Join a team focused on scalable, cost-efficient data platforms.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering or a similar discipline.
• Strong hands-on experience building and running production data pipelines on Databricks on Microsoft Azure with end-to-end delivery.
• Expert-level PySpark and Spark SQL, plus advanced SQL for analytics and ML readiness.
• Deep Delta Lake expertise, including ACID, MERGE/upsert, time travel and schema evolution.
• Proven track record delivering medallion data products and data models for analytics/ML.

Responsibilities

• Independently design, build and own production-grade data pipelines on Databricks.
• Model Bronze/Silver/Gold data products and curate data for analytics and ML.
• Engineer batch and streaming ingestion using Auto Loader and Structured Streaming.
• Tune Spark/Delta performance and cost; manage partitioning and clustering.
• Develop pipelines as software with tests, CI/CD and code reviews.

Skills

PySpark

Spark SQL

Delta Lake

Delta Live Tables

Databricks Workflows

Structured Streaming

Performance tuning

Python

CI/CD

Azure Databricks

Education

Bachelor's or Master's degree in Computer Science, Information Systems, Engineering

Tools

Auto Loader

Databricks Asset Bundles

Terraform

Azure DevOps

GitHub Actions

Delta Live Tables

Structured Streaming

Delta Lake

Job description

We are seeking an experienced Senior Databricks Engineer with deep, hands-on data engineering expertise to independently design, build and own production-grade pipelines on our Databricks Lakehouse. In this role you will develop complex transformations at scale in PySpark and Spark SQL, and deliver curated Bronze/Silver/Gold data products on the medallion architecture that power analytics, reporting and machine learning. You will own your pipelines end to end - design, build, test, deploy, optimize and support - working closely with data architecture, analytics, data science and platform engineering teams to deliver reliable, performant and cost-efficient data products across development, test and production.

Key Responsibilities & Skillsets:

• Independently design, build and own production-grade data pipelines on Databricks, from source ingestion through curated delivery, using PySpark, Spark SQL, Delta Live Tables and Databricks Workflows.
• Build and maintain Bronze/Silver/Gold data products on the medallion architecture - raw ingestion, cleansing and conformance, and business-ready aggregates modelled for analytics and ML consumption.
• Implement complex transformations at scale - multi-source joins, SCD Type 1/2 history, deduplication, late-arriving and out-of-order data, CDC merges, windowing and business-rule logic.
• Engineer batch and streaming ingestion from files, databases, APIs and event streams using Auto Loader, Structured Streaming, Delta Lake MERGE and change data feed.
• Tune Spark and Delta performance and cost - partitioning, liquid clustering, Z-ordering, OPTIMIZE and VACUUM, caching, broadcast strategy, skew and spill remediation, Photon and right-sized compute.
• Model curated data products with consuming teams - dimensional and star schemas, semantic layers, and serving through Databricks SQL warehouses and Delta Sharing.
• Engineer data quality and observability into every pipeline - expectations, schema enforcement and evolution, reconciliation and threshold checks, freshness and volume SLAs, and actionable alerting.
• Develop pipelines as software - modular Python packages, unit and integration tests, code review, Git branching, CI/CD with Azure DevOps or GitHub Actions, and deployment via Databricks Asset Bundles.
• Work within Unity Catalog governance - catalogs, schemas, volumes, lineage, tagging and fine-grained access - applying the standards set by the platform administration team.
• Own production support for your pipelines - triage, root cause analysis, backfills and reprocessing, and continuous hardening; mentor junior engineers and set code, design and documentation standards.

Candidate Profile:

• A Bachelor's or Master's degree in Computer Science, Information Systems, Engineering or a similar discipline.
• Strong hands-on experience building and running production data pipelines on Databricks on Microsoft Azure (AWS or GCP exposure a plus), with a proven ability to deliver independently end to end.
• Expert-level PySpark and Spark SQL, plus advanced SQL - window functions, complex joins, CTEs and performance-oriented query design.
• Deep Delta Lake expertise - ACID transactions, MERGE and upsert patterns, time travel, schema evolution, change data feed, OPTIMIZE, VACUUM and liquid clustering.
• Proven track record delivering Bronze/Silver/Gold (medallion) data products, including data modelling for analytics, BI and ML consumption.
• Hands-on experience with Databricks Workflows, Delta Live Tables, Auto Loader and Structured Streaming for batch and near-real-time pipelines.
• Strong Spark performance tuning and cost optimization skills - reading query plans and the Spark UI, and diagnosing skew, spill, shuffle and small-file problems.
• Production-grade software engineering practice - modular Python, testing with pytest, Git and code review, CI/CD (Azure DevOps or GitHub Actions), and Databricks Asset Bundles or Terraform.
• Working knowledge of Unity Catalog and the surrounding Azure data ecosystem - ADLS Gen2, Azure Data Factory, Key Vault, Event Hubs or Kafka, Synapse or Microsoft Fabric, and Microsoft Purview.
• Excellent problem-solving, documentation and stakeholder communication skills, with experience mentoring junior engineers; Databricks certifications (Data Engineer Associate/Professional) and Azure certifications (DP-203) are a plus.

What we offer:

• EXL Analytics offers an exciting, fast paced and innovative environment, which brings together a group of sharp and entrepreneurial professionals who are eager to influence business decisions. From your very first day, you get an opportunity to work closely with highly experienced, world-class analytics consultants.
• You can expect to learn many aspects of businesses that our clients engage in. You will also learn effective teamwork and time-management skills - key aspects for personal and professional growth
• Analytics requires different skill sets at different levels within the organization. At EXL Analytics, we invest heavily in training you in all aspects of analytics as well as in leading analytical tools and techniques.
• We provide guidance/ coaching to every employee through our mentoring program wherein every junior level employee is assigned a senior level professional as advisors.
• Sky is the limit for our team members. The unique experiences gathered at EXL Analytics sets the stage for further growth and development in our company and beyond.