Senior Databricks Engineer - End-to-End Data Pipelines

EXL; United States; On-site; USD 130,000 - 195,000; Full time

Benefits offered by this job

Mentoring program

Growth opportunities

Job summary

EXL is seeking a Senior Databricks Engineer to independently design, build and own production-grade data pipelines on the Databricks Lakehouse. You will deliver Bronze/Silver/Gold data products for analytics and ML, using PySpark, Spark SQL, and Delta Live Tables.

You will implement complex transformations, tune performance, and ensure observability and governance with Unity Catalog and Azure data services. Join a team focused on scalable, cost-efficient data platforms.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering or a similar discipline.
• Strong hands-on experience building and running production data pipelines on Databricks on Microsoft Azure with end-to-end delivery.
• Expert-level PySpark and Spark SQL, plus advanced SQL for analytics and ML readiness.
• Deep Delta Lake expertise, including ACID, MERGE/upsert, time travel and schema evolution.
• Proven track record delivering medallion data products and data models for analytics/ML.

Responsibilities

• Independently design, build and own production-grade data pipelines on Databricks.
• Model Bronze/Silver/Gold data products and curate data for analytics and ML.
• Engineer batch and streaming ingestion using Auto Loader and Structured Streaming.
• Tune Spark/Delta performance and cost; manage partitioning and clustering.
• Develop pipelines as software with tests, CI/CD and code reviews.

Skills

PySpark

Spark SQL

Delta Lake

Delta Live Tables

Databricks Workflows

Structured Streaming

Performance tuning

Python

CI/CD

Azure Databricks

Education

Bachelor's or Master's degree in Computer Science, Information Systems, Engineering

Tools

Auto Loader

Databricks Asset Bundles

Terraform

Azure DevOps

GitHub Actions

Delta Live Tables

Structured Streaming

Delta Lake

Job description

EXL is seeking a Senior Databricks Engineer to independently design, build and own production-grade data pipelines on the Databricks Lakehouse. You will deliver Bronze/Silver/Gold data products for analytics and ML, using PySpark, Spark SQL, and Delta Live Tables.

You will implement complex transformations, tune performance, and ensure observability and governance with Unity Catalog and Azure data services. Join a team focused on scalable, cost-efficient data platforms.