Data Engineer

Digitide; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Digitide Bengaluru seeks a skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure. You will enable data accessibility, quality, and insights across the organization using Azure Databricks, PySpark, and Data Factory.

Responsibilities include developing data models in Snowflake, optimizing performance, and collaborating with data scientists and analysts to deliver reliable data assets.

Qualifications

• Bachelor in CS or data engineering or related field.
• Proficiency in Azure Databricks for data processing.
• Strong PySpark and SQL skills and data modeling understanding.
• Ability to troubleshoot and optimize data workflows.

Responsibilities

• Data pipeline development: design, build, and optimize data pipelines to ingest, transform, and load data from multiple sources using Azure Databricks.
• Data architecture: develop and manage data models within Snowflake for efficient data organization and accessibility.
• Performance optimization: monitor and optimize pipeline performance and resolve issues.
• Collaboration: work with data scientists and analysts to provide reliable, well-structured data.

Skills

PySpark

SQL

Data modeling

Spark

Azure Databricks

Python

Scala

Hadoop ecosystem

Education

Bachelor's degree in CS/Data Engineering or related field

Tools

Azure Data Factory

Snowflake

Azure DevOps

GitHub

Job description

We are seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure. You will play a crucial role in our data ecosystem by working with cloud technologies to enable data accessibility, quality, and insights across the organization. This role requires expertise in Azure Databricks, PySpark (must have),Azure Data Factory, SQL, any cloud (preferably Azure)and Data Modelling.

• Bachelors in Computer Science, Data Engineering, or related field.
• Proficiency in Azure Databricks for data processing and pipeline orchestration.
• Strong Pyspark and SQL skills and understanding of data modeling principles.
• Ability to troubleshoot and optimize data workflows.

Key Responsibilities

• Data Pipeline Development: Design, build, and optimize data pipelines to ingest, transform, and load data from multiple sources, using Azure Databricks,
• Data Architecture: Develop and manage data models within Snowflake, ensuring efficient data organization and accessibility.
• Performance Optimization: Monitor and optimize pipeline performance, troubleshooting and resolving issues as needed.
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to support data-driven projects and provide access to reliable, well-structured data.

Strong Communication, stakeholder management skills. Strong in requirement gathering

Qualifications

• Having relevant Experience in MS Azure data bricks with Pyspark and Big Data Hadoop eco-system components
• Understanding of Hadoop Architecture and underlying framework including Storage Management.
• Strong understand and implementation experience in Spark, Databricks and ADF
• Expertise in implementing Data lake solution using Scala as well as Python.
• Expertise with orchestration tool like Azure Data Factory
• Strong SQL and Programing skills
• Experience with DataBricks is desirable
• Implementation experience with CICD tools such as Azure DevOps, GITHUB is desirable.