Data Engineer(Azure, ADF, Databricks, PySpark, Unity Catalog, SQL) || 6 to 10 Years || Bangalore

Innova ESI; Bengaluru; On-site; INR 800,000 - 1,500,000; Full time

Job summary

An established industry player is on the lookout for a talented Data Engineer to join their dynamic team. This role focuses on leveraging Azure technologies to design and optimize data pipelines while ensuring data integrity and security. You will work with cutting-edge tools like Azure Data Factory and Databricks, developing scalable ETL workflows and implementing best practices in data governance. If you are passionate about data engineering and eager to make a significant impact in a collaborative environment, this opportunity could be the perfect fit for you. Join a forward-thinking company that values innovation and excellence in the IT services sector.

Qualifications

• 6+ years of experience in Data Engineering with Azure technologies.
• Expertise in ADF, Databricks, and PySpark for ETL workflows.

Responsibilities

• Develop and optimize data pipelines using ADF and Databricks.
• Implement SCD2 in Delta Tables for effective data management.

Skills

Data Engineering

Azure Data Factory (ADF)

Azure Databricks

PySpark

SQL

Data Governance

ETL/ELT Workflows

Education

Bachelor's Degree in Computer Science or related field

Tools

Delta Tables

Unity Catalog

Azure Synapse Analytics

Power BI

Kafka

Job description

Data Engineer (Azure, ADF, Databricks, PySpark, Unity Catalog, SQL) || 6 to 10 Years || Bangalore

Job Description – Data Engineer

Role Overview:

Looking for a highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta Tables, Unity Catalog, Slowly Changing Dimension Type 2 (SCD2), and PySpark. The ideal candidate will be responsible for designing, developing, and optimizing data pipelines and ETL workflows while ensuring data integrity, scalability, and security within the Azure ecosystem.

Key Responsibilities:

• Develop and optimize data pipelines using Azure Data Factory (ADF) and Azure Databricks for large-scale data processing.
• Implement Slowly Changing Dimension in Delta Tables to manage historical data changes effectively.
• Leverage Unity Catalog for secure and organized data governance, cataloging, and access control across Databricks.
• Write efficient PySpark code to process and transform large datasets, ensuring high performance and scalability.
• Design and implement ETL/ELT solutions to integrate data from multiple sources into Delta Lake.
• Monitor, debug, and optimize existing data pipelines to ensure smooth operations and minimal downtime.
• Ensure data quality, consistency, and lineage tracking through best practices and automation.
• Collaborate with data architects, analysts, and business teams to define requirements and implement data-driven solutions.

Required Skills & Qualifications:

• 6+ years of experience in Data Engineering with a focus on Azure technologies.
• Expertise in Azure Data Factory (ADF) & Azure Databricks for ETL/ELT workflows.
• Strong knowledge of Delta Tables & Unity Catalog for efficient data storage and management.
• Experience with Slowly Changing Dimensions (SCD2) implementation in Delta Lake.
• Proficiency in PySpark for large-scale data processing & transformation.
• Hands-on experience with SQL & performance tuning for data pipelines.
• Understanding of data governance, security, and compliance best practices in Azure.
• Knowledge of CI/CD, DevOps practices for data pipeline automation.

Preferred Qualifications:

• Experience with Azure Synapse Analytics, Data Lakes, and Power BI integration.
• Knowledge of Kafka or Event Hub for real-time data ingestion.
• Certifications in Microsoft Azure (DP-203, DP-900) or Databricks are a plus.

Seniority Level: Mid-Senior level

Employment Type: Full-time

Job Function: Information Technology

Industries: IT Services and IT Consulting