Senior Data Engineer

Digital Omnis Consulting; Kuala Lumpur; On-site; MYR 120,000 - 180,000; Full time

Job summary

Digital Omnis Consulting is seeking a skilled Data Engineer in Kuala Lumpur to join the project team on a 1-year renewable contract. You will design, build, and manage cloud-based data pipelines and ETL processes to support analytics and reporting solutions, focusing on Azure and Databricks.

You will design data pipelines with Azure Data Factory, Databricks, and Python, while ensuring data quality, governance, and security standards are met.

Qualifications

• Experience designing, building, and maintaining data pipelines and ETL processes.
• Strong problem-solving skills and ability to work in an agile, collaborative environment.
• Proficiency in Python or similar languages used in data engineering.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Data Factory and Databricks.
• Build and optimize ETL processes to support data integration and analytics.
• Develop data workflows and scripts using Python, R, or similar languages.
• Implement DevOps methodologies for automated data pipeline deployment and monitoring.
• Collaborate with Cloud Architects and Platform Engineers to leverage Azure technologies.
• Ensure data quality, governance, and security standards are met.
• Support business teams with data engineering best practices and troubleshooting.

Skills

ETL

Agile teamwork

Tools

Azure Data Factory

Databricks

Shell (Linux)

Job description

We are seeking a skilled Data Engineer to join our project team on a 1-year renewable contract. The role involves designing, building, and managing cloud-based data pipelines and ETL processes to support advanced analytics and reporting solutions. The Data Engineer will work closely with project stakeholders to implement scalable data platforms primarily on Azure and Databricks.

Key Responsibilities:

• Design, develop, and maintain data pipelines using tools such as Azure Data Factory and Databricks.
• Build and optimize ETL processes to support data integration and analytics.
• Develop data workflows and scripts using Python, R, or similar languages.
• Implement DevOps methodologies for automated data pipeline deployment and monitoring.
• Collaborate with Cloud Architects and Platform Engineers to ensure solutions leverage best practices in cloud-based data platforms, with a focus on Azure technologies.
• Ensure data quality, governance, and security standards are met.
• Support business teams with data engineering best practices and troubleshooting.

Required Qualifications:

• Minimum 3 years of experience in data engineering with a strong background in data pipelines (e.g. Azure Data Factory, Databricks, or similar).
• Good experience with ETL design, development, and optimization.
• Proficiency in Python, Shell (Linux), or other programming languages suitable for data engineering.
• Practical experience with DevOps methodologies (CI/CD pipelines, version control, monitoring).
• Strong problem-solving skills and ability to work in an agile and collaborative environment.

Preferred Qualifications:

• Experience with Azure cloud technologies (Data Lake, Synapse, Azure SQL, etc.).
• Hands-on experience in Databricks ecosystem and its components.
• Familiarity with big data frameworks and tools.