Data Engineer(Azure, ADF, Databricks, PySpark, Unity Catalog, SQL) || 6 Years || Bangalore

Innova ESI; Bengaluru; On-site; INR 800,000 - 1,600,000; Full time

Job summary

An established industry player is on the lookout for a talented Data Engineer to join their innovative team. In this role, you will design and optimize data pipelines using cutting-edge Azure technologies, ensuring data integrity and scalability. You will leverage tools like Azure Data Factory and Databricks to create efficient ETL workflows while collaborating with cross-functional teams to implement data-driven solutions. This is a fantastic opportunity to work in a dynamic environment where your contributions will directly impact the company's data strategy and governance. If you are passionate about data engineering and eager to work with the latest technologies, this position is perfect for you.

Qualifications

• 6+ years of experience in Data Engineering with Azure technologies.
• Expertise in Azure Data Factory and Databricks for ETL workflows.

Responsibilities

• Develop and optimize data pipelines using Azure Data Factory and Databricks.
• Implement ETL/ELT solutions to integrate data into Delta Lake.

Skills

Data Engineering

Azure Data Factory (ADF)

Azure Databricks

PySpark

SQL

Data Governance

ETL/ELT Workflows

Slowly Changing Dimensions (SCD2)

Education

Bachelor's Degree in Computer Science or related field

Tools

Delta Tables

Unity Catalog

Azure Synapse Analytics

Power BI

Kafka

Event Hub

Job description

Job Description – Data Engineer (Azure, ADF, Databricks, PySpark, SCD, Unity Catalog, SQL)

Role Overview:

Looking for highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta Tables, Unity Catalog, Slowly Changing Dimension Type 2 (SCD2), and PySpark. The ideal candidate will be responsible for designing, developing, and optimizing data pipelines and ETL workflows while ensuring data integrity, scalability, and security within the Azure ecosystem.

Key Responsibilities:

• Develop and optimize data pipelines using Azure Data Factory (ADF) and Azure Databricks for large-scale data processing.
• Implement Slowly Changing Dimension in Delta Tables to manage historical data changes effectively.
• Leverage Unity Catalog for secure and organized data governance, cataloging, and access control across Databricks.
• Write efficient PySpark code to process and transform large datasets, ensuring high performance and scalability.
• Design and implement ETL/ELT solutions to integrate data from multiple sources into Delta Lake.
• Monitor, debug, and optimize existing data pipelines to ensure smooth operations and minimal downtime.
• Ensure data quality, consistency, and lineage tracking through best practices and automation.
• Collaborate with data architects, analysts, and business teams to define requirements and implement data-driven solutions.

Required Skills & Qualifications:

• 6+ years of experience in Data Engineering with a focus on Azure technologies.
• Expertise in Azure Data Factory (ADF) & Azure Databricks for ETL/ELT workflows.
• Strong knowledge of Delta Tables & Unity Catalog for efficient data storage and management.
• Experience with Slowly Changing Dimensions (SCD2) implementation in Delta Lake.
• Proficiency in PySpark for large-scale data processing & transformation.
• Hands-on experience with SQL & performance tuning for data pipelines.
• Understanding of data governance, security, and compliance best practices in Azure.
• Knowledge of CI/CD, DevOps practices for data pipeline automation.

Preferred Qualifications:

• Experience with Azure Synapse Analytics, Data Lakes, and Power BI integration.
• Knowledge of Kafka or Event Hub for real-time data ingestion.

Certifications in Microsoft Azure (DP-203, DP-900) or Databricks are a plus.

Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Information Technology

Industries

IT Services and IT Consulting