Azure Data Engineer

Deloitte Shared Services India; Coimbatore District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Deloitte Shared Services India is seeking an Azure Data Engineer to design and maintain scalable data pipelines using Databricks and Azure data services in a fast-paced analytics environment.

The role focuses on data integrity, governance, and collaboration with data science teams to enable robust BI and analytics solutions.

Qualifications

• Strong experience with Databricks for processing and analytics.
• Proficient in writing optimized SQL queries for data extraction and transformation.
• Hands-on PySpark experience for large-scale data processing.
• Deep understanding of ETL/ELT concepts and tooling.

Responsibilities

• Design, develop, and deploy scalable data pipelines using Databricks, ADF, and Azure data services.
• Implement ETL/ELT processes to ingest, transform, and load data into lakes and warehouses.
• Optimize data pipelines for performance and scalability; monitor data availability and integrity.
• Collaborate with data scientists and analysts to deliver data solutions.

Skills

Databricks

SQL

PySpark

ETL/ELT

Data Modeling

Data Governance

Tools

Azure Data Factory

Azure Data Lake Storage

Azure SQL Database

Key Vault

Job description

Job Description:

We are seeking a skilled and motivated Azure Data Engineer Years of experience to join our dynamic team. The ideal candidate will have a strong background in Databricks, Azure Data Services, and data engineering practices. You will be responsible for designing, developing, and maintaining scalable data pipelines, optimizing data workflows, and ensuring the integrity and availability of data for analytics and business intelligence.

Key Responsibilities:

1) Design and Development:

• Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL), Azure Data Factory, and other Azure data services.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
• Optimize and tune data pipelines for performance and scalability.

2) Data Processing:

• Write and optimize complex SQL queries for data extraction, transformation, and analysis.
• Use PySpark for large-scale data processing and analytics.
• Implement data partitioning, bucketing, and indexing strategies for efficient data retrieval.

3) Data Integration:

• Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
• Work with APIs, streaming data, and batch processing to ensure seamless data integration.

4) Data Governance and Quality:

• Implement data governance practices to ensure data quality, consistency, and security.
• Monitor and troubleshoot data pipelines to ensure data accuracy and availability.

5) Collaboration:

• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Work closely with DevOps teams to deploy and monitor data pipelines in production environments.

6) Documentation:

• Document data pipelines, workflows, and processes for knowledge sharing and future reference.
• Maintain up-to-date documentation on data architecture and data models.

Must-Have Skills:

• Databricks: Hands-on experience with Databricks for data processing, analytics, and Serverless SQLWH, Unity Catalog, Lakehouse, Medallion Architecture
• Azure Data Services: Proficiency in Azure Data Factory, Azure Data Lake Storage, and Azure SQL Database, Key Vault, Azure Pricing Model
• SQL: Strong expertise in writing and optimizing complex SQL queries.
• PySpark: Experience in using PySpark for data processing and transformation.
• ETL/ELT: Strong understanding of ETL/ELT processes and tools.
• Data Modeling: Knowledge of data modeling techniques and best practices.
• Data Governance: Familiarity with data governance, data quality, and data security practices.Role & responsibilities