Azure Data Engineer

UST USource; Bayan Lepas; On-site; MYR 180,000 - 300,000; Full time

Job summary

UST USource is seeking a highly skilled Lead Data Engineer to join our Data & Analytics team in Malaysia. You will design, develop, and manage end-to-end data solutions on Microsoft Azure, enabling reliable ingestion, transformation, storage, and analytics for business operations and decision making.

The role emphasizes strong hands-on experience with ADF, Synapse, Databricks, and data warehousing, with collaboration across business and technical teams to deliver scalable data platforms.

Qualifications

• 8+ years of hands-on experience with Azure Data Factory (ADF), including pipeline orchestration and Mapping Data Flows.
• Experience with Azure Synapse Analytics, ADLS Gen2, and Azure Databricks.
• Expertise in REST API integrations, authentication, pagination, and error handling.
• Strong knowledge of ETL/ELT, data pipeline design, incremental loading, and data validation.
• Experience in Data Warehousing, Dimensional Modeling, Star Schema, and Snowflake Schema.
• Good understanding of metadata-driven architecture and scalable cloud data platforms.

Responsibilities

• Design and develop end-to-end data pipelines using Azure Data Factory.
• Ingest data from REST APIs, SharePoint, Excel, and ADLS Gen2.
• Build scalable ETL/ELT solutions using ADF Mapping Data Flows and metadata-driven frameworks.
• Develop and optimize data models, data warehouses, and analytics solutions using Azure Synapse Analytics.
• Leverage Azure Databricks and Apache Spark for large-scale data processing.
• Implement data quality, validation, and monitoring processes to ensure reliable datasets.
• Collaborate with business and technical teams to deliver robust data solutions.
• Monitor, troubleshoot, and enhance production data pipelines.

Skills

ADF pipelines

Azure Synapse

ADLS Gen2

Azure Databricks

REST APIs

ETL/ELT design

Dimensional Modeling

Metadata-driven

Azure DevOps

Azure Functions

Tools

Azure Data Factory

Azure Synapse Analytics

ADLS Gen2

Azure Databricks

Job description

We are seeking a highly skilled and passionate Lead Data Engineer to join our growing Data & Analytics team. In this role, you will be responsible for designing, developing, and managing end-to-end data solutions on the Microsoft Azure platform. You will play a critical role in enabling reliable data ingestion, transformation, storage, and analytics capabilities that support business operations, reporting, and strategic decision‑making.

The ideal candidate will have strong experience in Azure Data Engineering technologies, data warehousing, ETL/ELT development, and cloud-native data architectures, with the ability to translate business requirements into scalable and efficient data solutions.

Key Responsibilities

• Design and develop end-to-end data pipelines using Azure Data Factory (ADF).
• Ingest data from REST APIs, SharePoint, Excel, and ADLS Gen2.
• Build scalable ETL/ELT solutions using ADF Mapping Data Flows and metadata-driven frameworks.
• Develop and optimize data models, data warehouses, and analytics solutions using Azure Synapse Analytics.
• Leverage Azure Databricks and Apache Spark for large-scale data processing.
• Implement data quality, validation, and monitoring processes to ensure reliable datasets.
• Collaborate with business and technical teams to deliver robust data solutions.
• Monitor, troubleshoot, and enhance production data pipelines.

Required Skills

• 8+ Years of Strong hands‑on experience with Azure Data Factory (ADF), including pipeline orchestration and Mapping Data Flows.
• Experience with Azure Synapse Analytics, ADLS Gen2, and Azure Databricks.
• Expertise in REST API integrations, authentication, pagination, and error handling.
• Strong knowledge of ETL/ELT, data pipeline design, incremental loading, and data validation.
• Experience in Data Warehousing, Dimensional Modeling, Star Schema, and Snowflake Schema.
• Good understanding of metadata-driven architecture and scalable cloud data platforms.

Preferred Skills

• Experience with Azure DevOps (CI/CD pipelines).
• Hands‑on exposure to Azure Functions and Azure Logic Apps.
• Knowledge of Microsoft Fabric and modern data architecture patterns.
• Azure certifications such as DP-203 are a plus.