Senior Data Engineer - Azure Databricks, DBT & Capital Markets

Multiverse Solutions; Dadri; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Multiverse Solutions is seeking a Senior Data Engineer with 10+ years of experience to design and implement scalable data platforms in Azure Databricks and dbt. You will develop robust ELT pipelines, apply Medallion Architecture with Delta Lake, and ensure data quality across capital markets datasets.

The role requires deep expertise in PySpark, SQL, and data governance, with a focus on performance and reliability in complex financial environments. On-site in India is expected.

Qualifications

• 10+ years of experience in data engineering or related field.

Responsibilities

• Design, develop, and optimize scalable data engineering solutions using Azure Databricks.
• Build and maintain robust ELT pipelines using dbt.
• Develop complex data transformations using PySpark, Spark SQL, and SQL.
• Implement Medallion Architecture (Bronze, Silver, Gold) with Delta Lake.
• Develop and manage dbt models, macros, snapshots, incremental models, and tests.
• Design data pipelines for Capital Markets data including trade, transaction, market, pricing, and reference data.
• Handle large financial datasets ensuring accuracy, consistency, and performance.
• Implement data quality checks, reconciliation, validation, and governance practices.
• Optimize Databricks workloads, Spark jobs, SQL queries, and dbt transformations.
• Implement CI/CD pipelines and version control using Git, GitHub, and Azure DevOps.
• Collaborate with solution architects and stakeholders to deliver enterprise-grade data solutions.

Skills

Azure Databricks

dbt

PySpark

Spark SQL

Delta Lake

ELT pipelines

Data modeling

Git / GitHub

Data governance

Capital Markets domain

Tools

GitHub

CI/CD

Azure Data Factory

Azure DevOps

Job description

Job Description: Senior Data Engineer Azure Databricks, dbt & Capital Markets

Overview

We are looking for a highly experienced Senior Data Engineer with 10+ years of experience in Data Engineering, specializing in Azure Databricks, dbt (Data Build Tool), and Capital Markets. The ideal candidate should have strong hands‑on experience designing and developing scalable data platforms, modern ELT pipelines, and data transformation solutions, preferably within the Capital Markets or Financial Services domain.

Key Responsibilities

1. Design, develop, and optimize scalable data engineering solutions using Azure Databricks.
2. Build and maintain robust ELT pipelines using dbt (Data Build Tool).
3. Develop complex data transformations using PySpark, Spark SQL, and advanced SQL.
4. Implement Medallion Architecture (Bronze, Silver, and Gold) using Delta Lake.
5. Develop and manage dbt models, macros, snapshots, incremental models, and automated testing frameworks.
6. Design and implement data pipelines for Capital Markets data, including trade, transaction, market data, pricing, and reference data.
7. Work with large and complex financial datasets, ensuring data accuracy, consistency, and performance.
8. Implement data quality checks, reconciliation, validation, and data governance practices.
9. Optimize Databricks workloads, Spark jobs, SQL queries, and dbt transformations.
10. Implement CI/CD pipelines and version control using Git, GitHub, and Azure DevOps.
11. Collaborate with solution architects, business stakeholders, and cross‑functional teams to deliver enterprise‑grade data solutions.

Required Qualifications & Skills

• 10+ years of overall experience in Data Engineering.
• Strong hands‑on experience with Azure Databricks and dbt.
• Excellent proficiency in PySpark, Spark SQL, and advanced SQL.
• Extensive experience in Delta Lake and Medallion Architecture.
• Strong experience developing dbt models, macros, snapshots, incremental models, and tests.
• Good understanding of modern ELT architecture and large‑scale data processing.
• Strong experience with data modeling, Star Schema, and Snowflake Schema.
• Hands‑on experience with Git, GitHub, CI/CD, and deployment practices.
• Strong understanding of data quality, data governance, and performance optimization.

Capital Markets Domain Experience

• Strong exposure to Capital Markets, Investment Banking, or Financial Services.
• Understanding of financial market data, trade lifecycle, transactions, securities, and pricing data.
• Experience handling market data, reference data, trade data, and portfolio‑related datasets.
• Knowledge of financial data processing, reconciliation, regulatory reporting, and risk analytics is preferred.

Preferred Skills

• Experience with Azure Data Factory or Apache Airflow.
• Knowledge of Azure Data Lake Storage Gen2.
• Experience with Unity Catalog and Databricks Workflows.
• Exposure to Azure DevOps and GitHub Actions.
• Experience working with financial data platforms and enterprise data warehouses.
• Relevant Databricks, dbt, or Microsoft Azure certifications.

Experience Required

10+ Years of Overall Data Engineering Experience

Core Technologies:

Azure Databricks, dbt, PySpark, Spark SQL, Advanced SQL, Delta Lake, Azure Data Factory, and Git.

Domain:

Capital Markets / Investment Banking / Financial Services.