Data Engineer (Azure & Databricks) - Unity Catalog

EY; Bengaluru; Hybrid; INR 2,400,000 - 4,000,000; Full time

Job summary

EY is seeking a Data Engineer to build and optimize end-to-end data pipelines using Python, PySpark, Databricks, and Azure cloud services in Bengaluru. The role involves developing a robust enterprise data warehouse and lakehouse solutions, plus collaboration with senior engineers on architectural design and best practices.

The ideal candidate has 8–10 years of data engineering experience, strong SQL skills, and hands-on Databricks experience in an Azure environment, with Git for version control

Qualifications

• 8 to 10 years of hands-on experience in Data Engineering, ETL development, or Data Analytics.
• Unity Catalog experience ranging between 3-4 years.
• Strong proficiency in Python for data manipulation and scripting.
• Solid understanding of relational databases and complex SQL queries, joins, and aggregations.
• Hands-on experience with Apache Spark (PySpark or Spark SQL) and the Databricks environment.
• Exposure to Azure data ecosystem, ADLS Gen2, and pipelines in Azure Data Factory.
• Foundational understanding of Data Warehousing principles (Star schema, facts & dimensions).
• Experience using Git for source code version control.

Responsibilities

• Pipeline Development: Build, test, and maintain ETL/ELT pipelines to move data into Azure Data Lake and Data Warehouse.
• Coding & Transformation: Write PySpark and Python code in Azure Databricks to clean and transform data.
• Data Integration: Orchestrate data movement with Azure Data Factory workflows.
• SQL Development: Write and optimize SQL queries for data extraction and reporting views.
• Monitoring & Support: Monitor daily batch jobs and troubleshoot failed pipelines.
• Collaboration: Work with senior engineers on architectures, code reviews, and best practices.
• Documentation: Maintain technical docs for pipelines, data dictionaries, and runbooks.

Skills

Python

SQL

PySpark

Databricks

Azure Cloud

Git

Data Warehousing concepts

Tools

Azure Data Factory

Azure Databricks

Delta Lake

ADLS Gen2

Azure DevOps / GitHub Actions

Job description

Job Title: Data Engineer (Azure & Databricks)

Experience Required:5-8 & 8 to 10 Years

About the Role

We are looking for a motivated and detail oriented Data Engineer to join our growing analytics team. In this role, you will work closely with Lead Data Engineers and Architects to develop, maintain, and optimize data pipelines using modern cloud technologies.

You will get hands on experience building our Enterprise Data Warehouse and Lakehouse solutions using Python, Databricks, and Microsoft Azure. This is an excellent opportunity for an emerging data professional to grow their skills in cloud data engineering and distributed computing.

Key Responsibilities

• . Pipeline Development: Build, test, and maintain robust ETL/ELT pipelines to move data from various sources into our Azure Data Lake and Data Warehouse.
• . Coding & Transformation: Write clean, efficient PySpark and Python code in Azure Databricks to clean, transform, and aggregate data.
• . Data Integration: Utilize Azure Data Factory (ADF) to orchestrate data movement and schedule automated workflows.
• . SQL Development: Write and optimize SQL queries for data extraction, analysis, and creating views for downstream reporting tools.
• . Monitoring & Support: Monitor daily batch jobs, troubleshoot failed pipelines, and resolve data quality issues in a timely manner.
• . Collaboration: Work alongside senior engineers to understand architectural designs, participate in code reviews, and learn enterprise best practices for data engineering.
• . Documentation: Maintain clear technical documentation for data pipelines, data dictionaries, and operational runbooks.

Required Skills & Qualifications

• . Experience: 8 to 10 years of hands on experience in Data Engineering, ETL development, or Data Analytics.
• . Unity Catalog experience ranging between 3-4 years
• . Python Programming: Strong proficiency in Python for data manipulation (e.g., using Pandas) and scripting.
• . SQL Mastery: Solid understanding of relational databases and the ability to write complex SQL queries, joins, and aggregations.
• . Big Data / Databricks: Hands on experience with Apache Spark (PySpark or Spark SQL) and familiarity with the Databricks environment.
• . Azure Cloud: Exposure to the Azure data ecosystem, specifically working with Azure Data Lake Storage (ADLS Gen2) and creating pipelines in Azure Data Factory (ADF).
• . Data Warehousing Concepts: Good foundational understanding of Data Warehousing principles (Star schema, facts, and dimensions).
• . Version Control: Experience using Git for source code version control.

Preferred/Bonus Skills

• . Understanding of Delta Lake concepts (Time travel, ACID transactions).
• . Familiarity with CI/CD concepts and deploying code using Azure DevOps or GitHub Actions.
• . Prior experience working in an Agile/Scrum environment.
• . Certifications: Azure Data Fundamentals (DP 900) or Databricks Certified Data Engineer Associate.