Data Engineer

Quest Global; Bengaluru; Hybrid; INR 1,200,000 - 2,400,000; Full time

Job summary

Quest Global is seeking an experienced Data Engineer to drive end-to-end data solution design for large-scale use cases in Bengaluru. The role emphasizes Azure SQL, ADLS, Cosmos DB, and Databricks, with strong Python data munging skills.

You will build and optimize data pipelines across IaaS/PaaS/SaaS, leverage Azure DevOps for deployment, and engage with ETL tools like Informatica, DataStage, and SSIS where relevant. Experience in regulated environments is a plus.

Qualifications

• 3 to 6+ years of experience of Data Engineering roles including 2+ years in leading end-to-end design and implementation of data solutions for large scale use cases.
• Experience in designing solutions using databases and data storage technology using Azure data management components - Azure SQL, ADLS, Cosmos
• Experience building, optimizing and automating data pipelines, architectures and data sets using MS Azure data management and processing components through IaaS/PaaS/SaaS implementation models
• Proficiency in Python using different modules used for data munging
• Be up to date with data processing technology / platforms such as Spark (Databricks)
• Experienced in Azure DevOps
• Desirable: Experience of working in tightly regulated industry is desirable
• Experience with ETL and/or data integration tool such as Informatica, DataStage, SSIS is highly desirable
• Good understanding of Azure infrastructure components and their fit in different types of data solutions

Responsibilities

• Lead end-to-end design and implementation of data solutions for large-scale use cases.
• Design solutions using Azure SQL, ADLS and Cosmos DB.
• Build, optimize and automate data pipelines using MS Azure components through IaaS/PaaS/SaaS models.
• Munge and transform data with Python using various libraries.
• Work with Spark (Databricks) for data processing tasks.
• Collaborate with Azure DevOps for deployment and automation.
• Familiarity with ETL tools like Informatica, DataStage, SSIS is advantageous.
• Understand Azure infrastructure considerations for different data solutions.

Skills

Python

Data pipelines

Azure DevOps

Tools

Azure SQL

ADLS

Cosmos DB

Databricks

Informatica

DataStage

SSIS

Job description

Role & responsibilities

3 to 6+ years of experience of Data Engineering roles including 2+ years in leading end-to-end design and implementation of data solutions for large scale use cases

• Experience in designing solutions using databases and data storage technology using Azure data management components - Azure SQL, ADLS, Cosmos
• Experience building, optimizing and automating data pipelines, architectures and data sets using MS Azure data management and processing components through IaaS/PaaS/SaaS implementation models implemented through custom solutions
• Proficiency in Python using different modules used for data munging
• Be up to date with data processing technology / platforms such as Spark (Databricks)
• Experienced in Azure DevOps
• Desirable
• Experience of working in tightly regulated industry is desirable
• Experience with ETL and/or data integration tool such as Informatica, DataStage, SSIS is highly desirable
• Good understanding of Azure infrastructure components and their fit in different types of data solutions

Preferred candidate profile