Data Engineer

HABIB Group; Selangor; On-site; MYR 120,000 - 150,000; Full time

Job summary

A prominent data solutions company based in Malaysia seeks an experienced Data Engineer to design and maintain data pipelines and develop data lakes. The ideal candidate will have 6+ years of experience, strong skills in SQL and Python, and a proven track record in building production data platforms. The role involves working with hybrid environments and ensuring the security and performance of data solutions. Excellent communication skills are essential for collaboration with cross-functional teams.

Qualifications

• Bachelor’s degree or equivalent experience.
• 6+ years of hands‑on experience in data engineering.
• Proven experience building and supporting production data platforms.
• Strong skills in SQL and Python; experience with Apache Spark.
• Able to build datasets for BI tools such as Power BI, Tableau, or Qlik.

Responsibilities

• Design, build, and maintain data pipelines.
• Develop and manage data lake platforms.
• Implement scalable ETL solutions.
• Ensure data solutions are secure and optimized.
• Work closely with data scientists to deliver datasets.

Skills

Data engineering

SQL proficiency

Python programming

Apache Spark

Data lakes and warehouses

Data security

Communication skills

Education

Bachelor's degree in Computer Science, IT, Engineering, or related field

Tools

Databricks

Power BI

Tableau

Git

Azure DevOps

Job description

Overview

Design, build, and maintain data pipelines for ingesting, transforming, and integrating data from multiple sources (databases, APIs, files, streaming).

Develop and manage data lake / lakehouse platforms to support analytics, BI, and AI/ML use cases.

Design and support hybrid data environments, combining on-premise systems with cloud platforms (primarily Azure).

Implement scalable ETL/ELT solutions for batch and real-time data processing.

Set up and manage analytics and ML workspaces (e.g. Databricks, notebooks) for team collaboration.

Ensure data solutions are secure, reliable, well-monitored, and optimized for performance and cost.

Implement data quality checks, validation, and monitoring to maintain high data reliability.

Apply data governance and security best practices, including access control, encryption, and metadata/lineage management.

Work closely with data scientists, analysts, and business stakeholders to deliver analytics-ready datasets.

Coordinate with system and data vendors to onboard new data sources and maintain stable integrations.

Translate business requirements into maintainable, production-ready data solutions.

Write high-quality Python, SQL, and Spark code, following engineering best practices.

Support DataOps / MLOps initiatives, including CI/CD, testing, version control, and pipeline observability.

Requirementsh3> Requirements:

Bachelor’s degree in Computer Science, IT, Engineering, or related field (or equivalent experience).

6+ years of hands‑on experience in data engineering, owning data pipelines end‑to‑end.

Proven experience building and supporting production data platforms.

Experience working with hybrid environments (on‑premise and cloud data systems).

Strong skills in SQL and Python; experience with Apache Spark is a must.

Good understanding of data lakes, data warehouses, and lakehouse architectures.

Hands‑on with relational and NoSQL databases; exposure to cloud data warehouses is a plus.

Experience with real‑time/event streaming and data pipeline orchestration tools.

Knowledge of data security, access control, encryption, and compliance.

Experience implementing data quality checks, validation, and testing.

Familiar with Git and CI/CD pipelines (Azure DevOps, GitHub Actions) and basic Infrastructure‑as‑Code.

Able to build high‑performance datasets for BI tools such as Power BI, Tableau, or Qlik.

Exposure to machine learning pipelines and supporting batch or real‑time inference is an advantage.

Strong communication skills with the ability to explain technical concepts clearly.

Independent, well‑organized, and able to work effectively with cross‑functional teams and vendors.