Data Engineer

UST USource; Bayan Lepas; On-site; MYR 75,000 - 95,000; Full time

Job summary

A global technology firm is seeking a skilled Azure Data Engineer to design and maintain data solutions on the Microsoft Azure platform. In this role, you will develop ETL/ELT pipelines using services like Azure Data Factory and Databricks, ensuring data quality and performance. The ideal candidate will have over 4 years of data engineering experience with a strong focus on Azure services. This position is crucial for driving data-driven initiatives across various business units within the firm.

Qualifications

• Minimum 4+ years of experience in data engineering or related roles.
• Minimum 2+ years of hands-on experience with Azure data solutions.
• Solid experience with Azure Core Data Services.

Responsibilities

• Design, develop, test, and deploy ETL/ELT data pipelines using Azure.
• Monitor data pipeline execution and optimize performance.
• Collaborate with analysts and data scientists for technical solutions.

Skills

SQL

Azure Data Factory

Azure Databricks

Python

Spark (PySpark or Scala)

Data Warehousing

Git

Education

Bachelor's degree in Computer Science or related field

Tools

Azure Data Lake Storage

Azure Synapse Analytics

Job description

We are seeking a skilled Azure Data Engineer to join a Data & Analytics team. This role focuses on designing, developing, implementing, and maintaining scalable and reliable data solutions on the Microsoft Azure platform. The Azure Data Engineer will build and optimize data pipelines, ensure data quality, and collaborate with analysts, data scientists, and other stakeholders to support data-driven initiatives.

Responsibilities

• Design, develop, test, and deploy ETL/ELT data pipelines using Azure services such as Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.
• Implement data transformations, data cleaning, aggregation, and enrichment logic according to business requirements.
• Build and maintain data structures and models within Azure Data Lake Storage and Azure Synapse Analytics (or other data warehousing solutions).
• Monitor data pipeline execution, troubleshoot failures, and optimize performance and cost-effectiveness.
• Implement data quality checks and processes to ensure data accuracy and reliability.
• Collaborate with data analysts, data scientists, and business users to understand data requirements and translate them into technical solutions.
• Work closely with senior engineers and architects, contributing to solution design and adhering to established best practices.
• Create and maintain clear technical documentation for data pipelines and processes.

Qualifications

• Bachelor\'s degree in Computer Science, Engineering, Information Technology, or a related quantitative field (or equivalent practical experience).
• Minimum 4+ years of experience in data engineering, business intelligence, or related roles involving data integration and transformation.
• Minimum 2+ years of hands-on experience developing and implementing data solutions specifically on the Microsoft Azure platform.
• Core Azure Data Services: Solid experience with Azure Data Factory (ADF), Azure Synapse Analytics (SQL/Spark Pools), Azure Databricks, and Azure Data Lake Storage (ADLS Gen2).
• Data Processing & Programming: Strong proficiency with SQL is essential. Good proficiency with Spark (using PySpark or Scala) and Python (preferred) or Scala for data manipulation.
• Data Concepts: Understanding of data warehousing concepts, dimensional modeling, ETL/ELT patterns, and data lake principles.
• Databases: Experience with relational databases (e.g., Azure SQL Database). Familiarity with NoSQL databases is a plus.
• Version Control: Experience using Git for code management.