Sr. Azure Data Engineer

Noventiq; Dadri; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Noventiq is seeking a highly skilled data professional to manage end-to-end analytics workloads using Microsoft Fabric and Azure tools in Dadri, Uttar Pradesh, India. The successful candidate will design and develop scalable data processing solutions with a focus on high-performance reporting and orchestration.

Key responsibilities include building multi-stage data pipelines, implementing Medallion Architecture formats, and leveraging Python and SQL for complex transformations. Deep experience with Azure Data Factory and Databricks is essential for driving analytics initiatives.

Qualifications

• Extensive hands‑on experience with Microsoft Fabric and Azure Data Factory.
• Deep proficiency in Python and PySpark for data processing.
• Expert-level SQL scripting for data manipulation and query optimization.

Responsibilities

• Design and develop scalable data processing engines using Azure Databricks.
• Architect multi-stage data pipelines using Azure Data Factory.
• Implement Medallion Architecture using Delta Lake formats.

Skills

Microsoft Fabric

Azure Databricks

Azure Synapse Analytics

Azure Data Factory

Python

PySpark

SQL

Delta Lake

OneLake

Git

Job description

Required Skills and Qualification

• Primary Toolset: Extensive hands‑on experience with Microsoft Fabric, Azure Databricks, Azure Synapse Analytics, and Azure Data Factory.
• Expert Scripting: Python: Deep proficiency in Python and PySpark (Spark Core, Spark SQL, Structured Streaming).
• Advanced SQL: Expert‑level SQL scripting (Window functions, CTEs, stored procedures, and query optimization).
• Delta Lake & OneLake: Strong understanding of Delta Lake table formats and the unified storage principles of OneLake.
• Orchestration Patterns: Proven ability to build reusable, parameter‑driven pipelines in ADF that handle diverse data sources (REST APIs, Parquet, Delta).
• DevOps & CI/CD: Good to have experience with Git‑based version control (Azure DevOps/GitHub) and deploying data infrastructure as code (Bicep/Terraform).

Roles and Responsibilities

• Experience in building and managing end‑to‑end analytics workloads using Microsoft Fabric, OneLake, Lakehouses, and Warehouses. Implement Direct Lake Connectivity for high‑performance Power BI reporting.
• Experience design and develop scalable data processing engines using Azure Databricks. Leverage PySpark for complex transformations, streaming, and large‑scale data wrangling.
• Experience in architecting multi‑stage data pipelines using Azure Data Factory (ADF) and Synapse Pipelines. Focus on metadata‑driven frameworks and dynamic orchestration to minimize hard‑coding.
• Advanced Scripting & Transformation:
  • SQL Scripts: Write high‑performance T‑SQL and Spark SQL for complex business logic, data validation, and performance tuning in Synapse Dedicated/Serverless pools.
  • Python: Develop custom Python modules for API integrations, automation scripts, and advanced data manipulation beyond standard ETL tools.
• Experience in implementing the Medallion Architecture (Bronze/Silver/Gold) using Delta Lake formats to ensure ACID transactions, data lineage, and schema evolution.