Senior Data Engineer - Databricks & DBT

Mployee.me; Dadri; Remote; INR 4,000,000 - 7,000,000; Full time

Job summary

Mployee.me in Noida is seeking a Senior Data Engineer with 10+ years of experience to design and implement scalable data pipelines using Databricks and DBT, with strong Python, PySpark, and SQL expertise.

You will lead DBT models, work with Delta Lake and Lakehouse architectures on cloud platforms (AWS/Azure/GCP), optimize performance, implement CI/CD for data projects, and mentor junior engineers while collaborating with architects and analysts.

Qualifications

• 10+ years of overall experience in Data Engineering.
• Strong hands-on experience with Databricks and DBT.
• Expertise in developing and managing large-scale data pipelines.

Responsibilities

• Design, develop, and implement scalable end-to-end data pipelines using Databricks and DBT.
• Lead the development of data transformation workflows using DBT, including models, macros, snapshots, tests, and incremental models.
• Develop and optimize complex data processing solutions using Python, PySpark, and SQL.
• Work extensively with Databricks, Delta Lake, and Lakehouse architecture.
• Design and implement robust ETL/ELT processes for large-scale structured and unstructured datasets.
• Build and maintain data integration solutions across multiple data sources.
• Optimize Databricks workloads, Spark jobs, SQL queries, and DBT models for performance and cost efficiency.
• Implement data quality checks, validation frameworks, testing, and monitoring.
• Work with cloud platforms such as AWS, Azure, or GCP.
• Implement CI/CD pipelines, version control, and deployment strategies for data engineering projects.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and cross-functional teams.
• Provide technical leadership, mentor junior engineers, and conduct code reviews.
• Troubleshoot production issues and ensure data pipeline reliability and availability.

Skills

Databricks

DBT

Python

PySpark

SQL

ETL/ELT

Cloud

Delta Lake

Lakehouse

Data Warehousing

Tools

Airflow

Azure Data Factory

GitHub

Azure DevOps

Snowflake

Redshift

Job description

Job Title: Senior Data Engineer - Databricks & DBT
Experience: 10+ Years
Primary Skills: Databricks, DBT, Python, PySpark, SQL
Contact: [e-mail]
Location: Noida

Job Summary

We are looking for a highly experienced Senior Data Engineer with 10+ years of experience in designing, developing, and implementing scalable data engineering solutions. The ideal candidate must have strong hands-on expertise in Databricks and DBT (Data Build Tool), along with advanced knowledge of Python, PySpark, SQL, ETL/ELT pipelines, and cloud-based data platforms.

Key Responsibilities

1. Design, develop, and implement scalable end-to-end data pipelines using Databricks and DBT.
2. Lead the development of data transformation workflows using DBT, including models, macros, snapshots, tests, and incremental models.
3. Develop and optimize complex data processing solutions using Python, PySpark, and SQL.
4. Work extensively with Databricks, Delta Lake, and Lakehouse architecture.
5. Design and implement robust ETL/ELT processes for large-scale structured and unstructured datasets.
6. Build and maintain data integration solutions across multiple data sources.
7. Optimize Databricks workloads, Spark jobs, SQL queries, and DBT models for performance and cost efficiency.
8. Implement data quality checks, validation frameworks, testing, and monitoring.
9. Work with cloud platforms such as AWS, Azure, or GCP.
10. Implement CI/CD pipelines, version control, and deployment strategies for data engineering projects.
11. Collaborate with Data Architects, Data Scientists, Business Analysts, and cross-functional teams.
12. Provide technical leadership, mentor junior engineers, and conduct code reviews.
13. Troubleshoot production issues and ensure data pipeline reliability and availability.

Required Technical Skills

• Databricks
• Strong hands-on experience
• DBT
• Advanced experience in DBT Core / DBT Cloud
• Python
• Advanced programming skills
• PySpark
• Strong development experience
• SQL
• Advanced SQL and query optimization
• Delta Lake
• Delta tables, Lakehouse architecture
• ETL/ELT
• End-to-end pipeline development
• Cloud
• AWS / Azure / GCP
• Data Warehousing
• Snowflake / Azure Synapse / Redshift
• Orchestration
• Airflow / Azure Data Factory
• CI/CD
• Git, GitHub, Azure DevOps

Mandatory Requirements

• 10+ years of overall experience in Data Engineering.
• Strong hands-on experience with Databricks and DBT.
• Expertise in developing and managing large-scale data pipelines.
• Strong proficiency in Python, PySpark, and advanced SQL.
• Experience with DBT models, macros, snapshots, testing, documentation, and incremental materializations.
• Strong understanding of Databricks architecture, Delta Lake, and Lakehouse concepts.
• Experience with cloud-based data engineering solutions.
• Proven experience in performance optimization and production support.
• Experience leading technical teams and handling end-to-end data engineering projects.