Data Engineer

Blue Data Digital SDN BHD; Kuala Lumpur; On-site; MYR 120,000 - 180,000; Full time

Job summary

Blue Data Digital SDN BHD in Kuala Lumpur is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines and platforms. You will own end-to-end data flows, from ingestion to modeling and delivery, enabling analytics, BI, and AI/ML initiatives across the organization.

The ideal candidate has hands-on Python and SQL expertise, strong data modeling knowledge, and the ability to architect solutions on cloud platforms (GCP preferred).

Qualifications

• Bachelor's degree in a relevant field (CS/IT/Data Science).
• Strong Python and SQL skills with hands-on data engineering experience.
• Experience building batch and real-time data pipelines.
• Familiarity with data warehousing concepts and data modeling.

Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines using ETL/ELT patterns.
• Ingest and integrate structured, semi-structured, and unstructured data from diverse sources.
• Implement incremental loading, CDC, and idempotent pipeline design.
• Design and optimize data models for analytics and reporting.
• Build and manage data warehouses and lakehouses (BigQuery, Cloud Storage).
• Orchestrate workflows using Cloud Composer, Dagster, Prefect, or similar tools.
• Implement data validation and quality checks (Great Expectations, dbt tests, Soda, Dataplex).
• Monitor, troubleshoot, and optimize pipelines for performance and cost.
• Collaborate with analytics, BI, AI/ML, and software teams to meet business needs.

Skills

Python

SQL

Data modeling

ETL/ELT

Cloud platforms

Data pipelines

Education

Bachelor's degree in Computer Science or related field

Tools

Cloud Composer

Dagster

Prefect

dbt

Soda

Dataplex

BigQuery

Cloud Storage

Job description

We are looking for an experienced Data Engineer to design, build, and maintain scalable, reliable, and high-performance data platforms. You will own data pipelines end to end, from ingestion and transformation to modeling, quality, and delivery, enabling analytics, business intelligence, and AI/ML initiatives across the organization. The ideal candidate has strong hands‑on expertise in Python and SQL, deep knowledge of modern data engineering practices, and the ability to independently architect and deliver data solutions on cloud platforms, with Google Cloud Platform (GCP) preferred.

Key responsibilities

Design, build, and maintain scalable batch and real-time data pipelines using ETL/ELT patterns.

Ingest and integrate structured, semi-structured, and unstructured data from diverse sources, including APIs, databases, files, and event streams.

Implement incremental loading, change data capture (CDC), and idempotent pipeline design.

Design and optimize data models (dimensional modeling, star/snowflake schemas, Data Vault where relevant) for analytics and reporting.

Build and manage data warehouses and lakehouses following the Medallion architecture (Bronze, Silver, Gold layers), ideally on BigQuery and Cloud Storage.

Orchestrate workflows using tools such as Cloud Composer (Apache Airflow), Dagster, Prefect, or other cloud-native schedulers.

Implement data validation, testing, and quality checks using tools such as Great Expectations, dbt tests, Soda, or Dataplex data quality.

Monitor, troubleshoot, and optimize pipelines for performance, cost, and reliability.

Build and manage data solutions on cloud platforms, preferably GCP, using services such as BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, Cloud Composer, and Cloud Run.

Partner with analytics, BI, AI/ML, and software engineering teams to deliver data that meets business needs.

About you

3+ years of hands‑on experience in data engineering.

Strong proficiency in Python for data processing and pipeline development.

Advanced SQL skills, including complex joins, window functions, CTEs, query optimization, and performance tuning.

Proven experience building and maintaining ETL/ELT pipelines in production.

Strong understanding of data modeling, data warehousing concepts, and database design.

Experience with relational databases (PostgreSQL, MySQL, SQL Server) and familiarity with NoSQL databases (MongoDB, Cassandra, DynamoDB, Firestore, Bigtable).

Hands‑on experience with at least one cloud platform and its data services; GCP experience is preferred.

Strong troubleshooting, analytical, and problem‑solving skills.

Bachelor's degree in computer science, Information Technology, Data Science, Engineering, or a related field (or equivalent practical experience).