Senior Data Engineer - Databricks & DBT

Multiverse Solutions; Dadri; On-site; INR 4,500,000 - 6,000,000; Full time

Job summary

Multiverse Solutions is seeking a Senior Data Engineer (Databricks & DBT) with 10+ years of experience to design and implement scalable data pipelines. You will work with Databricks, DBT, Python, PySpark, and SQL, building robust end-to-end ETL/ELT solutions in a Lakehouse environment.

This role involves optimizing workloads, ensuring data quality, and mentoring junior engineers while collaborating with data architects, scientists, and analysts.

Qualifications

• Databricks and DBT hands-on experience
• 10+ years of data engineering experience
• Strong Python, PySpark, and SQL proficiency
• Experience with Delta Lake and Lakehouse
• Experience building scalable end-to-end data pipelines
• Cloud platforms AWS/Azure/GCP

Responsibilities

• Design end-to-end data pipelines with Databricks and DBT
• Lead data transformation workflows using DBT
• Develop data processing with Python, PySpark, SQL
• Work with Databricks, Delta Lake, Lakehouse
• Implement robust ETL/ELT processes at scale
• Build data integrations across sources
• Optimize workloads and queries for performance
• Implement data quality checks and monitoring
• Work with AWS, Azure, or GCP
• Implement CI/CD and version control for data projects
• Collaborate with architects, scientists, analysts
• Provide technical leadership and code reviews
• Troubleshoot production issues and ensure reliability

Skills

Databricks

DBT

Python

PySpark

SQL

Delta Lake

ETL/ELT

Cloud

Data Warehousing

Orchestration

CI/CD

Git

Tools

DBT Core/DBT Cloud

Airflow

Azure Data Factory

Snowflake

Azure Synapse

Redshift

AWS

GCP

Job description

Job Title: Senior Data Engineer - Databricks & DBT
Experience: 10+ Years
Primary Skills: Databricks, DBT, Python, PySpark, SQL
Contact: [e-mail]
Location: Noida

Job Summary

We are looking for a highly experienced Senior Data Engineer with 10+ years of experience in designing, developing, and implementing scalable data engineering solutions. The ideal candidate must have strong hands-on expertise in Databricks and DBT (Data Build Tool), along with advanced knowledge of Python, PySpark, SQL, ETL/ELT pipelines, and cloud-based data platforms.

Key Responsibilities

1. Design, develop, and implement scalable end-to-end data pipelines using Databricks and DBT.
2. Lead the development of data transformation workflows using DBT, including models, macros, snapshots, tests, and incremental models.
3. Develop and optimize complex data processing solutions using Python, PySpark, and SQL.
4. Work extensively with Databricks, Delta Lake, and Lakehouse architecture.
5. Design and implement robust ETL/ELT processes for large-scale structured and unstructured datasets.
6. Build and maintain data integration solutions across multiple data sources.
7. Optimize Databricks workloads, Spark jobs, SQL queries, and DBT models for performance and cost efficiency.
8. Implement data quality checks, validation frameworks, testing, and monitoring.
9. Work with cloud platforms such as AWS, Azure, or GCP.
10. Implement CI/CD pipelines, version control, and deployment strategies for data engineering projects.
11. Collaborate with Data Architects, Data Scientists, Business Analysts, and cross-functional teams.
12. Provide technical leadership, mentor junior engineers, and conduct code reviews.
13. Troubleshoot production issues and ensure data pipeline reliability and availability.

Required Technical Skills

• Databricks
• Strong hands-on experience
• DBT
• Advanced experience in DBT Core / DBT Cloud
• Python
• Advanced programming skills
• PySpark
• Strong development experience
• SQL
• Advanced SQL and query optimization
• Delta Lake
• Delta tables, Lakehouse architecture
• ETL/ELT
• End-to-end pipeline development
• Cloud
• AWS / Azure / GCP
• Data Warehousing
• Snowflake / Azure Synapse / Redshift
• Orchestration
• Airflow / Azure Data Factory
• CI/CD
• Git, GitHub, Azure DevOps

Mandatory Requirements

• 10+ years of overall experience in Data Engineering.
• Strong hands-on experience with Databricks and DBT.
• Expertise in developing and managing large-scale data pipelines.
• Strong proficiency in Python, PySpark, and advanced SQL.
• Experience with DBT models, macros, snapshots, testing, documentation, and incremental materializations.
• Strong understanding of Databricks architecture, Delta Lake, and Lakehouse concepts.
• Experience with cloud-based data engineering solutions.
• Proven experience in performance optimization and production support.
• Experience leading technical teams and handling end-to-end data engineering projects.