Data Engineer (Data Bricks) Onsite

PradeepIT Consulting Services Pvt Ltd; Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A leading IT consulting firm in Bengaluru is seeking a skilled Data Engineer to design and build data pipelines using Spark-SQL and PySpark in Azure Databricks. The role involves collaboration with the global Analytics team and leading project initiatives. Candidates must have strong skills in Pyspark, Azure, ADF, and ETL processes. This position is onsite and offers an opportunity to work on innovative data solutions.

Responsibilities

• Design and build data pipelines using Spark-SQL and PySpark in Azure Databricks.
• Design and build ETL pipelines using ADF.
• Build and maintain a Lakehouse architecture in ADLS / Databricks.
• Perform data preparation tasks including data cleaning, normalization, deduplication, type conversion etc.
• Work with DevOps team to deploy solutions in production environments.
• Control data processes and take corrective action when errors are identified.
• Participate as a full member of the global Analytics team.
• Collaborate with Data Science and Business Intelligence colleagues.
• Lead projects that include other team members.
• Apply change management tools to manage upgrades and data migrations.

Skills

Pyspark

Azure

ADF

Databricks

ETL

SQL

Job description

Overview

Data Engineer (Data Bricks) Onsite

Must Have Skills

• Pyspark
• Azure
• ADF
• Databricks
• ETL
• SQL

Nice to Have Skills

• Change Management tool
• DevOps

Job Description

• Design and build data pipelines using Spark-SQL and PySpark in Azure Databricks
• Design and build ETL pipelines using ADF
• Build and maintain a Lakehouse architecture in ADLS / Databricks
• Perform data preparation tasks including data cleaning, normalization, deduplication, type conversion etc.
• Work with DevOps team to deploy solutions in production environments.
• Control data processes and take corrective action when errors are identified.
• Corrective action may include executing a work around process and then identifying the cause and solution for data errors.
• Participate as a full member of the global Analytics team, providing solutions for and insights into data related items.
• Collaborate with your Data Science and Business Intelligence colleagues across the world to share key learnings, leverage ideas and solutions and to propagate best practices.
• You will lead projects that include other team members and participate in projects led by other team members.
• Apply change management tools including training, communication and documentation to manage upgrades, changes and data migrations .