Data Engineer

Oxydata Software; Kuala Lumpur; On-site; MYR 80,000 - 120,000; Full time

Job summary

A leading company is seeking a (Senior) Data Engineer to enhance their global data platform. The role involves developing data pipelines, collaborating with international teams, and utilizing technologies like Azure and Databricks to support IoT factory data processing.

Qualifications

• 5+ years of experience in Data Engineering with a focus on ETL and data warehousing.
• Proficiency in Databricks and strong programming skills in Python.

Responsibilities

• Develop and maintain robust data pipelines for IoT data.
• Contribute to platform improvements including automation and data governance.

Skills

Data Engineering

ETL

Data Integration

Data Warehousing

Python

Communication

Tools

Databricks

Apache Spark

Azure

Docker

MongoDB

Airflow

Git

Job description

Job Title:

(Senior) Data Engineer

Location: Malaysia

Overview:

ROCKWOOL is seeking a (Senior) Data Engineer to join our global Data Science & Engineering team. Based in Malaysia, you will play a critical role in supporting and developing our global data platform, which collects and processes IoT factory data across our international operations. The current platform is built on Azure and leverages Databricks, Airflow, Docker, Blob Storage, and MongoDB. Future initiatives include migrating to a modern architecture involving Azure EventHub, Databricks, and Kafka.

You will be part of the operational arm of a global team, collaborating closely with colleagues in Poland and Denmark, as well as local team members in Malaysia consisting of Data Engineers, ML Engineers, and Data Scientists.

Key Responsibilities:

• Develop and maintain robust data pipelines for ingesting and transforming factory and IoT data.
• Contribute to platform improvements including automation, CI/CD processes, data governance, and monitoring.
• Address technical challenges and implement scalable and efficient features.
• Support upcoming migrations to new technologies such as Databricks, Azure EventHub, and Kafka.
• Collaborate across distributed teams and communicate effectively with global stakeholders.
• Document solutions and promote knowledge sharing across teams.

Required Qualifications:

Technical Skills:

• 5+ years of experience in Data Engineering, with a focus on ETL, data integration, and data warehousing.
• Proficiency in Databricks and Apache Spark.
• Solid experience working with Azure cloud services.
• Strong programming skills in Python.
• Familiarity with DevOps/DataOps practices.

Communication:

• Strong communication skills in English (C1 level).

Personal Attributes:

• Self-motivated and capable of independently driving tasks to completion.
• Proactive in identifying and proposing effective solutions.
• Strong collaboration skills with both technical and non-technical stakeholders.

Preferred Qualifications (Big Advantages):

• Experience with MongoDB or other NoSQL databases.
• Knowledge of Airflow or similar workflow orchestration tools.
• Proficiency in Docker and containerization practices.
• Familiarity with advanced Databricks features like Unity Catalog or Delta Live Tables (DLT).
• Strong SQL expertise and experience with both on-prem and cloud database management.
• Understanding of programming fundamentals, algorithms, and data structures.
• Experience with Git (preferably GitHub) and version control best practices.
• Solid grasp of modern data architecture concepts: data lakes, data vaults, and warehouses.
• Working knowledge of Infrastructure-as-Code (IaC) tools, preferably Terraform.
• Familiarity with productivity-enhancing tools like ChatGPT and GitHub Copilot.

Nice to Have:

• Experience with data modeling.
• Proficiency in additional languages such as C#, Java, Scala.
• Familiarity with DBT, Snowflake, or Kafka.

Other Expectations:

• Experience working with Scrum/Kanban methodologies and tools like Jira.
• Ability to gather requirements independently and work closely with business users.
• Proven experience working in international, cross-functional teams.