Azure Bigdata Specialist

PradeepIT Consulting Services Pvt Ltd; Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A tech consulting firm in Bengaluru seeks a Data Engineer with experience in Azure Databricks, ETL/ELT pipelines, and real-time data processing. The ideal candidate should have 4 to 8 years of IT experience, strong skills in Azure tools, and knowledge of CI/CD processes. This role offers an opportunity to work on large datasets and complex processing tasks in an innovative environment.

Qualifications

• 4 to 8 years of experience in IT Industry.
• Minimum 4 years of experience in Data Engineering using Azure tools.
• Experience with building ETL/ELT pipelines for large data sets.

Responsibilities

• Build and maintain ETL/ELT pipelines for large data sets.
• Real-time data processing using Kafka/EventHub.
• Infrastructure management and CI/CD pipelines.

Skills

Azure Databricks

Delta Lake

Azure Data Lake Storage

Apache Kafka

CI/CD

Python

SparkSQL

Education

BE, MS, M.Tech or MCA

Tools

Terraform

Docker

Kubernetes

Azure DevOps

Job description

Overview

Overall, 4 to 8 years of experience in IT Industry. Min 4 years of experience working on Data Engineering using Azure Databricks, Synapse, ADF/Airflow. At least 3 Project experience in Building and maintaining ETL / ELT pipelines for large data sets, complex data processing, transformations, business logics, cost monitoring & performance optimization, and feature engineering processes.

Responsibilities

• 3+ project experience in building and maintaining ETL / ELT pipelines for large data sets, complex data processing, transformations, business logic, cost monitoring & performance optimization, and feature engineering.

Must Have skills

• Extensive experience with Azure Databricks (ADB), Delta Lake, Azure Data Lake Storage (ADLS), Azure Data factory (ADF), Azure SQL Database (SQL DB), SQL, ELT / ETL Pipeline Development in Spark based environment.
• Extensive Experience with SparkCore, PySpark, Python, SparkSQL, Scala, Azure Blob Storage.
• Experience in Real-Time Data Processing using Apache Kafka/EventHub/IoT, Structured Streaming and Stream analytics.
• Experience with Apache Airflow for ELT Orchestration.
• Experience with infrastructure management, Infrastructure as code (e.g. Terraform)
• Experience with CI/CD, Version control tools like GitHub, Azure DevOps.
• Experience with Version control tools, building CI/CD pipelines.
• Experience with Azure cloud platform.

Good to have

• Experience / Knowledge on Containerizations - Docker, Kubernetes
• Experience working in Agile Methodology

Qualifications

• Qualifications - BE, MS, M.Tech or MCA