Senior Azure Data Engineer

Vidushi Infotech; United States; Remote; USD 130,000 - 170,000; Full time

Job summary

Vidushi Infotech is seeking a Senior Azure Data Engineer with 5–7 years of hands-on experience building and managing enterprise‑grade data pipelines on the Microsoft Azure stack. The role focuses on SQL, Python, PySpark, and Azure Databricks to design high‑performance ETL/ELT pipelines.

You will modernize legacy assets, ensure data quality and governance, and collaborate with data architects, analysts, and stakeholders.

Qualifications

• 5 to 7 years of hands-on professional experience building, managing, and optimizing scalable data pipelines on Azure.
• Expertise in SQL, Python, and PySpark for large‑scale data transformations.
• Proficient with data governance, testing, and data quality practices.

Responsibilities

• ETL/ELT design, development, and deployment of scalable pipelines using Azure Data Factory, Azure Databricks, and core Azure services.
• Write, test, and maintain production-grade code in Python, PySpark, and SQL.
• Modernize legacy data pipelines to improve throughput, maintainability, and reliability.

Skills

SQL

Python

PySpark

Azure Databricks

ADF

Azure Synapse

Git

CI/CD

Tools

Azure Data Factory (ADF)

Azure Databricks

Azure DevOps

GitHub

Git

Job description

Role Overview

We are seeking a highly skilled and experienced Senior Azure Data Engineer to join our dynamic data engineering team. The ideal candidate brings 5 to 7 years of hands‑on professional experience building, managing, and optimizing scalable, enterprise‑grade data pipelines within the Microsoft Azure ecosystem.

In this role, you will leverage deep expertise in SQL, Python, and PySpark alongside Azure Databricks to architect high‑performing ETL/ELT pipelines, modernize legacy data assets, and deliver reliable data products for analytics and downstream consumption.

Immediate Joiners preferred

PF Service History is Mandatory

Key Responsibilities

Pipeline Architecture & Development

• ETL/ELT Design: Design, build, and deploy robust, scalable data ingestion and transformation pipelines using Azure Data Factory (ADF), Azure Databricks, and core Azure services.
• Core Development: Write, test, and maintain optimized, production‑ready code using Python, PySpark, and SQLfor large‑scale batch and complex data transformations.
• Platform Modernization: Modernize and refactor legacy data pipelines to improve data throughput, code maintainability, and overall reliability.

Performance & Optimization

• Performance Tuning: Optimize PySpark jobs, SQL queries, and Databricks clusters to drastically reduce execution runtime and cloud computing costs.
• Quality & Governance: Write unit/integration tests and establish data validation frameworks to guarantee end‑to‑end data integrity and precision.

Modern Azure & Fabric Ecosystem Integration

• Azure Integration: Integrate data solutions across the Azure ecosystem, including Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Synapse Analytics, and Azure Key Vault.
• Microsoft Fabric: Apply working knowledge of Microsoft Fabric to help streamline and unify enterprise analytics, data engineering, and AI workloads.

Collaboration & DevOps

• CI/CD & DevOps: Implement modern software engineering principles, maintaining version control via Git (Azure DevOps / GitHub) and automated deployment pipelines.
• Cross-Functional Partnering: Partner closely with data architects, analysts, data scientists, and business stakeholders to translate complex business needs into technical specifications.

Required Qualifications

Experience

• 57 years of dedicated experience in Data Engineering, with a heavy focus on the Microsoft Azure data stack.

Mandatory Technical Skills

• SQL: Expert‑level mastery of advanced SQL (window functions, query plan optimization, complex joins, indexing, and partitioning strategies).
• Python & PySpark: Advanced proficiency in Python and PySpark for big data processing, data manipulation, and parallel processing execution.
• Azure Databricks & ADF: Proven hands‑on experience designing and orchestrating data pipelines using Azure Databricks (Spark SQL / PySpark) and Azure Data Factory.

Core Azure Infrastructure

• Strong practical experience working with Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Synapse Analytics (or Azure SQL DW).
• Hands‑on knowledge or working experience with Microsoft Fabric architecture and unified analytics patterns.

Software Engineering & CI/CD

• Proficient with Git workflows, unit testing frameworks (e.g., pytest), and CI/CD automated deployment practices within Azure DevOps or GitHub.
• Demonstrated ability to profile, troubleshoot, and optimize resource‑heavy Spark jobs and SQL queries.

Preferred / Bonus Qualifications

• Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203).
• Streaming Data: Experience handling real‑time streaming data with Azure Event Hubs, Azure Stream Analytics, or Apache Kafka.
• Data Security & Governance: Familiarity with Microsoft Purview, fine‑grained access control (RBAC), data masking, and security protocols in Azure.