Lead Azure Data Engineer: Craft Scalable Data Pipelines

UST USource; Bayan Lepas; On-site; MYR 180,000 - 300,000; Full time

Job summary

UST USource is seeking a highly skilled Lead Data Engineer to join our Data & Analytics team in Malaysia. You will design, develop, and manage end-to-end data solutions on Microsoft Azure, enabling reliable ingestion, transformation, storage, and analytics for business operations and decision making.

The role emphasizes strong hands-on experience with ADF, Synapse, Databricks, and data warehousing, with collaboration across business and technical teams to deliver scalable data platforms.

Qualifications

• 8+ years of hands-on experience with Azure Data Factory (ADF), including pipeline orchestration and Mapping Data Flows.
• Experience with Azure Synapse Analytics, ADLS Gen2, and Azure Databricks.
• Expertise in REST API integrations, authentication, pagination, and error handling.
• Strong knowledge of ETL/ELT, data pipeline design, incremental loading, and data validation.
• Experience in Data Warehousing, Dimensional Modeling, Star Schema, and Snowflake Schema.
• Good understanding of metadata-driven architecture and scalable cloud data platforms.

Responsibilities

• Design and develop end-to-end data pipelines using Azure Data Factory.
• Ingest data from REST APIs, SharePoint, Excel, and ADLS Gen2.
• Build scalable ETL/ELT solutions using ADF Mapping Data Flows and metadata-driven frameworks.
• Develop and optimize data models, data warehouses, and analytics solutions using Azure Synapse Analytics.
• Leverage Azure Databricks and Apache Spark for large-scale data processing.
• Implement data quality, validation, and monitoring processes to ensure reliable datasets.
• Collaborate with business and technical teams to deliver robust data solutions.
• Monitor, troubleshoot, and enhance production data pipelines.

Skills

ADF pipelines

Azure Synapse

ADLS Gen2

Azure Databricks

REST APIs

ETL/ELT design

Dimensional Modeling

Metadata-driven

Azure DevOps

Azure Functions

Tools

Azure Data Factory

Azure Synapse Analytics

ADLS Gen2

Azure Databricks

Job description

UST USource is seeking a highly skilled Lead Data Engineer to join our Data & Analytics team in Malaysia. You will design, develop, and manage end-to-end data solutions on Microsoft Azure, enabling reliable ingestion, transformation, storage, and analytics for business operations and decision making.

The role emphasizes strong hands-on experience with ADF, Synapse, Databricks, and data warehousing, with collaboration across business and technical teams to deliver scalable data platforms.