Databricks Developer / Data Architect

EXL Talent Acquisition Team; Pune District; Hybrid; INR 1,200,000 - 1,600,000; Full time

Job summary

EXL Talent Acquisition Team is seeking a Databricks Developer / Data Architect to design, implement, and optimize modern data platforms using Databricks and cloud-native tools. Responsibilities include building scalable ETL pipelines, establishing Lakehouse Federation, and enabling high-performance analytics across Azure and AWS.

The ideal candidate will have 5+ years in data engineering/architecture, hands-on Databricks expertise, and strong data modeling skills.

Qualifications

• 5+ years of experience in Data Engineering / Data Architecture.
• Strong hands-on experience with Databricks platform.
• Proficiency in Python, Spark, PySpark, and SQL.
• Knowledge of Lakehouse Architecture and Medallion Architecture.
• Cloud experience with Azure and AWS.
• Experience with CI/CD and DevOps practices.

Responsibilities

• Design scalable enterprise data architectures using Databricks Lakehouse platform.
• Configure Databricks workspaces, clusters, access controls, and governance.
• Implement Medallion Architecture (Bronze, Silver, Gold) for data processing.
• Set up Lakehouse Federation and data connectors for multi-source integration.
• Develop logical and physical data models for structured and semi-structured data.
• Ensure data quality, security, scalability, and performance optimization.
• Develop and maintain scalable ETL/ELT pipelines using PySpark and Spark SQL.
• Optimize Spark jobs for performance and cost efficiency.
• Integrate data from relational and NoSQL databases and cloud platforms.
• Automate deployment and monitoring of ETL workflows.

Skills

Python

Apache Spark

PySpark

SQL

Lakehouse Architecture

Medallion Architecture

Data Modeling

ETL/ELT Design Patterns

Azure

AWS

Tools

Databricks

Snowflake

PostgreSQL

MongoDB

DynamoDB

Cloudera

Domino Server

Job description

Job Description: Databricks Developer / Data Architect

Position Title

Databricks Developer / Data Architect

Location

Hybrid/ Remote

Employment Type

Full-Time / Contract

Job Summary

We are seeking an experienced Databricks Developer / Data Architect to design, implement, and optimize modern data platforms and ETL pipelines using Databricks and cloud-native technologies. The ideal candidate will have strong expertise in data architecture, Lakehouse implementation, Medallion Architecture, and scalable ETL development across Azure and AWS environments.

The role involves setting up Databricks workspaces, configuring data integrations and Lakehouse Federation, building enterprise-grade ETL workflows, and enabling high-performance analytics solutions.

Key Responsibilities

Data Architecture & Modeling

• Design and implement scalable enterprise data architectures using Databricks Lakehouse platform
• Configure and manage Databricks workspaces, clusters, access controls, and governance
• Implement Medallion Architecture (Bronze, Silver, Gold layers) for data processing and analytics
• Set up and manage Lakehouse Federation and data connectors for multi-source integration
• Develop logical and physical data models for structured and semi-structured datasets
• Ensure data quality, security, scalability, and performance optimization

ETL Development

• Develop and maintain scalable ETL/ELT pipelines using PySpark, Spark SQL, and Databricks workflows
• Build reusable data ingestion frameworks for batch and streaming workloads
• Optimize Spark jobs for performance, cost efficiency, and reliability
• Integrate data from relational and NoSQL databases, cloud platforms, and external systems
• Automate deployment and monitoring of ETL workflows

Cloud & Platform Engineering

• Work with Azure and AWS cloud services to deploy and manage data solutions
• Configure integrations with Snowflake, Postgres, MongoDB, DynamoDB, Cloudera, and Domino Server
• Support CI/CD, infrastructure automation, and environment management
• Collaborate with cross-functional teams including Data Scientists, Analysts, and Business stakeholders

Required Skills & Qualifications

• 5+ years of experience in Data Engineering / Data Architecture
• Strong hands-on experience with Databricks platform
• Expertise in:
  • Python
  • Apache Spark
  • PySpark
  • SQL
• Strong understanding of:
  • Lakehouse Architecture
  • Medallion Architecture
  • Data Modeling
  • ETL/ELT Design Patterns
• Experience with cloud platforms:
  • Microsoft Azure
  • AWS
• Experience integrating with:
  • PostgreSQL
  • DynamoDB
  • MongoDB
  • Snowflake
  • Cloudera
  • Domino Server
• Knowledge of performance tuning and optimization in Spark/Databricks
• Experience with version control and DevOps practices

Preferred Qualifications

• Databricks Certification(s)
• Experience with Delta Lake and Unity Catalog
• Familiarity with streaming frameworks and real-time data processing
• Knowledge of data governance and security best practices
• Experience in Agile/Scrum delivery models

Technical Stack

• Databricks
• Python
• Spark / PySpark
• SQL
• Azure
• AWS
• Snowflake
• PostgreSQL
• MongoDB
• DynamoDB
• Cloudera
• Domino Server

Soft Skills

• Strong analytical and problem-solving abilities
• Excellent communication and stakeholder management skills
• Ability to work independently and collaboratively in fast-paced environments
• Strong documentation and solution design capabilities

Nice-to-Have

• Experience with Terraform or Infrastructure as Code
• Exposure to ML/data science platforms
• Experience with orchestration tools such as Airflow or Azure Data Factory