Data Engineer (Databricks)

PradeepIT Consulting Services Pvt Ltd; Bengaluru; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A consulting services firm in Bengaluru is looking for a Data Engineer (Databricks) with 5-8 years of experience. The role involves designing data pipelines in Spark-SQL and PySpark, collaborating with global teams, and leading projects. Candidates should possess strong knowledge of Azure Databricks and ETL processes. This role offers opportunities for professional growth and collaboration in a dynamic environment.

Qualifications

• 5-8 years of experience in data engineering.
• Extensive knowledge of Spark-SQL and PySpark.
• Experience with Azure Databricks and ADF.

Responsibilities

• Design and build data pipelines using Spark-SQL and PySpark.
• Construct ETL pipelines using Azure Data Factory.
• Collaborate with global data teams on projects.

Job description

About the job Data Engineer (Databricks)

Data Engineer (Databricks) 5-8 years | Bangalore

Job Description

• Design and build data pipelines using Spark-SQL and PySpark in Azure Databricks
• Design and build ETL pipelines using ADF
• Build and maintain a Lakehouse architecture in ADLS / Databricks
• Perform data preparation tasks including data cleaning, normalization, deduplication, type conversion, etc.
• Work with DevOps team to deploy solutions in production environments
• Control data processes and take corrective action when errors are identified. Corrective action may include executing a workaround process and then identifying the cause and solution for data errors
• Participate as a full member of the global Analytics team, providing solutions for and insights into data related items
• Collaborate with your Data Science and Business Intelligence colleagues across the world to share key learnings, leverage ideas and solutions and to propagate best practices
• You will lead projects that include other team members and participate in projects led by other team members
• Apply change management tools including training, communication and documentation to manage upgrades, changes and data migrations

Must Have Skills: