Lead I - Data Engineering

UST; Maharashtra; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

UST in Maharashtra is seeking an experienced Azure Data Engineer (5+ years) to design, build, and maintain modern data pipelines using ADF, Databricks, Synapse, and ADLS Gen2. You will connect to SQL Server, REST APIs, and various data sources to deliver reliable data platforms for analytics and BI.

The role emphasizes code quality, CI/CD automation, data security with RBAC and Key Vault, and collaboration with analysts, architects, and data scientists to drive scalable, compliant data solutions.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Azure Data Engineer Associate (DP-203) certification preferred.
• 5+ years of experience in Azure Data Engineering.

Responsibilities

• Design, develop, and maintain Azure data pipelines with ADF for batch and incremental ingestion.
• Connect to SQL Server, REST APIs, CSV/JSON, SAP, and other sources; ingest into Azure.
• Develop metadata-driven pipelines for reusability.
• Implement data validation, error handling, and logging in ADF.
• Use Azure Databricks (PySpark) for cleansing, transformation, enrichment.
• Optimize Spark jobs for performance and cost.
• Implement ETL/ELT with Delta Lake and Bronze-Silver-Gold architecture.
• Work with ADLS Gen2 for raw and curated zones.
• Develop data models in Azure Synapse Analytics / SQL Server for reporting.
• Apply partitioning, indexing, and performance tuning.
• CI/CD pipelines via Azure DevOps or GitHub Actions for data workflows.
• Collaborate with architects to automate deployments with Git.
• Manage data access using Azure RBAC, Managed Identities, Key Vault.
• Ensure data security, compliance, privacy per standards.
• Collaborate with analysts, BI developers to define data requirements.
• Participate in code reviews and follow best practices.

Skills

Python

SQL

PySpark

Spark SQL

REST APIs

ETL

Data Modeling

Education

Bachelor's degree in Computer Science/IT

Azure Data Engineer Associate (DP-203)

Tools

Azure Data Factory

Azure Databricks

Azure Synapse

ADLS Gen2

Azure SQL Database

Git

Azure DevOps

Power BI

Delta Lake

Delta Live Tables

Synapse Pipelines

Job description

Role Description

Role Overview: We are seeking a skilled Azure Data Engineer with 5+ years of experience in designing, developing, and maintaining modern data pipelines and data integration solutions using Azure services. The ideal candidate should have strong expertise in Azure Data Factory (ADF), Azure Databricks, Azure Synapse, and Azure Data Lake Storage (ADLS). You will work closely with business analysts, architects, and data scientists to deliver reliable and scalable data solutions that power analytics and business intelligence platforms.

Key Responsibilities

• Data Ingestion & Integration Design, build, and maintain data pipelines using Azure Data Factory for batch and incremental data ingestion.
• Connect to various data sources (SQL Server, REST APIs, CSV, JSON, SAP, etc.) and integrate into the Azure ecosystem.
• Develop metadata-driven and parameterized pipelines to improve reusability.
• Implement data validation, error handling, and logging frameworks in ADF.
• Data Transformation & Processing Use Azure Databricks (PySpark) for data cleansing, transformation, and enrichment.
• Optimize Spark jobs for performance and cost efficiency.
• Implement ETL/ELT workflows with Delta Lake and Medallion (Bronze, Silver, Gold) architecture.
• Data Storage & Modeling Work with Azure Data Lake Storage Gen2 for raw and curated data zones.
• Develop data models in Azure Synapse Analytics / SQL Server for reporting and analytics.
• Implement partitioning, indexing, and performance tuning strategies.
• Deployment & DevOps Implement CI/CD pipelines using Azure DevOps or GitHub Actions for data workflows.
• Collaborate with architects to automate deployments and version control using Git.
• Security & Governance Manage data access using Azure RBAC, Managed Identities, and Key Vault.
• Ensure data security, compliance, and privacy as per organizational standards.
• Collaboration Work with data analysts, BI developers, and business users to define data requirements.
• Participate in code reviews and adhere to best practices in data engineering.

Technical Skills Required

• Category Skills Azure Services: Azure Data Factory, Azure Databricks, Azure Synapse, ADLS Gen2, Azure SQL Database
• Programming: Python (PySpark), SQL, Spark SQL
• Data Modeling: Star/Snowflake schema, Dimensional modeling
• Source Systems: SQL Server, Oracle, SAP, Flat Files (CSV, JSON, XML), REST APIs
• Version Control & CI/CD: Git, Azure DevOps
• Scheduling & Monitoring: ADF triggers, Databricks jobs, Log Analytics
• Security: Managed Identity, Key Vault, Access Control
• Preferred: Power BI basics, exposure to DataBricks Delta Live Tables or Synapse Pipelines
• Soft Skills: Strong analytical and problem-solving skills. Good communication and collaboration abilities. Ability to work in agile/scrum environments. Self-driven and proactive in identifying process improvements.

Educational Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Azure Data Engineer Associate (DP-203) certification preferred.

Example Project Responsibilities

• Design and implement end-to-end data ingestion from on-prem SQL Server to Azure Data Lake using ADF.
• Build Databricks notebooks for data cleansing and transformations using PySpark.
• Implement Delta Lake tables and load curated data into Synapse for reporting.
• Collaborate with BI teams to publish Power BI dashboards on top of Synapse datasets.

Optional (Good to Have)

• Experience with Real-time data processing (Event Hub / Stream Analytics).
• Knowledge of Infrastructure as Code (IaC) using Terraform or ARM templates.
• Familiarity with data quality and data catalog tools (Purview).

Skills

• Apache Spark
• Data Engineering
• Python
• SQL
• ETL
• PySpark