Azure Data Engineer

Straive; Hyderabad, Gurugram District, Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Straive is seeking a Senior Data Engineer to architect and implement a modern data platform. You will lead a transition toward a unified data experience, leveraging Azure for high-scale processing and Microsoft Fabric for integrated data engineering, with a heavy focus on automated data quality.

Key responsibilities include lakehouse architecture with Databricks and Fabric, governance with Purview, and building robust ETL/ELT pipelines using PySpark and Spark SQL to enable ML and BI workloads.

Qualifications

• SQL queries with complex joins and window functions.
• Advanced PySpark development with modular code and tuning.
• Extensive Databricks experience (Unity Catalog, Delta Live Tables).
• Strong Azure cloud skills with Azure Databricks, Fabric, Synapse & ADF.
• Experience with Azure Purview for governance and lineage.
• Big Data tech including Spark, Delta Lake, Structured Streaming.
• Understanding of Data Warehouse/Lake/Lakehouse architectures.

Responsibilities

• Lakehouse implementation using Databricks and Fabric Data Engineering.
• Governance and cataloging with Microsoft Purview.

Skills

SQL expertise

Python/PySpark

Databricks

Azure Cloud

Purview

Big Data

Data Warehouses

Tools

Databricks Unity Catalog

Delta Live Tables

Databricks Workflows

Fabric

Azure Synapse

Azure Data Factory

Purview

Job description

Role Overview

We are looking for a Senior Data Engineer to architect and implement our next-generation data platform. You will lead the transition toward a unified data experience, leveraging Azure for high-scale processing and Microsoft Fabric for integrated data engineering, with a heavy emphasis on automated data quality.

Key Responsibilities

• Lakehouse Implementation: Build and optimize Medallion architectures using Databricks and Fabric Data Engineering (Lakehouses and Notebooks).
• Governance & Cataloging: Implement data discovery, lineage, and classification using Microsoft Purview.
• Hybrid Orchestration: Design complex workflows using Azure Data Factory and Synapse Pipelines (Good to have).
• Data Transformation: Develop highly efficient ETL processes using PySpark and Spark SQL.
• High-Performance ETL: Architect and maintain robust ETL/ELT processes using Delta Live Tables (DLT) and Spark SQL, ensuring data is ready for downstream ML and BI workloads.
• Performance Engineering: Deep-dive into Spark UI and query plans to eliminate bottlenecks, optimize shuffles, and reduce cloud compute costs.

Technical Requirements

• SQL: Expert-level complex joins, window functions, and query optimization.
• Python/PySpark: Advanced capability in writing modular, unit-tested PySpark code and tuning Spark UI/Shuffle partitions.
• Deep experience in Databricks (Unity Catalog, Delta Live Tables, Databricks Workflows/Lakeflow).
• Deep expertise with Azure cloud with Azure Databricks, Fabric, Synapse & ADF.
• Experience with Azure Purview for enterprise-grade data governance.
• Big Data: Proficiency with technologies like Spark, Delta Lake, and Structured Streaming.
• Strong understanding of Data Warehouse/Lake/Lakehouse architecture and concepts.
• Programming: Strong Python/PySpark skills for data engineering tasks.

Good to have :

• DP-600/DP-700 - Microsoft Certified: Fabric Analytics Engineer
• Associate / Microsoft Certified: Fabric Data Engineer Associate
• DP-203 - Data Engineering on Microsoft Azure
• Databricks Certified Data Engineer

• Exposure of Databricks AI agent and AgentBricks