Data Engineer (Data Lake, Data Warehouse & MDM)

CMC-APAC PRIVATE LIMITED; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

CMC-APAC PRIVATE LIMITED is seeking an experienced Data Engineer to support enterprise data initiatives across data governance, data lakes, and data warehouses in hybrid cloud environments. You will design and optimize large pipelines and collaborate with data scientists and business teams.

The role requires 5+ years in data engineering, experience with MDM, Delta Lake/Databricks, and SQL/NoSQL databases. Strong problem-solving, communication, and ability to operate in a fast-paced,

Qualifications

• At least 5 years of experience as a Data Engineer.
• Experience building and operating large-scale data lakes and data warehouses.
• Experience working on Master Data Management projects.
• Proven ability to support and work with cross-functional teams in a dynamic environment.

Responsibilities

• Create and manage a single master record for each business entity, ensuring data consistency, accuracy, and reliability.
• Implement data governance processes, including data quality management, data profiling, data remediation, and automated data lineage.
• Create and maintain robust data processing pipelines across cloud, on-premises, and hybrid environments.
• Assemble large, complex data sets from diverse sources.
• Collaborate with Data Scientists, ML Engineers, and business users to derive actionable insights on data quality.

Skills

Data governance

MDM

Data modeling

Data quality

SQL

Python

Java

Scala

Spark

Kafka

ETL

CI/CD

Scrum

TDD

Tools

Delta Lake

Databricks

Hortonworks/Data Platform

Cloudera Data Platform

Informatica MDM

Talend

Semarchy xDM

IBM PIM/IKC

Profisee

Hive

HBase

Postgres

Spark Streaming

Job description

About the Role

We are looking for an experienced Data Engineer to support enterprise data initiatives involving master data management, data governance, data lakes, data warehouses, and large-scale data processing platforms. The role focuses on building and enhancing data pipelines, managing complex datasets, and supporting data-driven initiatives across cloud, on-premises, and hybrid environments.

Key Responsibilities

• Create and manage a single master record for each business entity, ensuring data consistency, accuracy, and reliability.
• Implement data governance processes, including data quality management, data profiling, data remediation, and automated data lineage.
• Create and maintain multiple robust and high-performance data processing pipelines within cloud, private data centre, and hybrid data ecosystems.
• Assemble large, complex data sets from a wide variety of data sources.
• Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and business users to derive actionable insights on data quality.
• Design and implement internal processes to automate manual workflows, optimize data delivery, and re-design infrastructure for greater scalability.
• Support and work with cross-functional teams in a dynamic environment.

Requirements

Experience

• At least 5 years of experience in a Data Engineer role.
• Experience building and operating large-scale data lakes and data warehouses.
• Experience working on projects for Master Data Management.
• Proven ability to support and work with cross-functional teams in a dynamic environment.

Data Platforms & Architecture

• Familiarity with data lake, data warehouse, and data lakehouse architectures.
• Familiarity with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality.
• Experience with Delta Lake and Databricks.
• Experience working with Hortonworks Data Platform or Cloudera Data Platform.

Data Governance & MDM

• Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage.
• Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee.
• Experience with Metadata Management tools.
• Exposure to Data Governance processes and tools.

Big Data & Streaming Technologies

• Experience with Hadoop ecosystem and big data tools, including Spark and Kafka.
• Experience with stream-processing systems including Spark Streaming.

Database & Query Optimization

• Advanced working experience with relational SQL and NoSQL databases, including Hive, HBase, and Postgres.
• Deep understanding of SQL and the ability to optimize data queries.
• Successful experience manipulating, processing, and extracting value from large, disconnected datasets.

Programming & Development

• Experience with object-oriented and scripting languages such as Python, Java, and Scala.
• Experience applying modern development principles, including Scrum, Test-Driven Development (TDD), Continuous Integration (CI), Code Reviews

Cloud & ETL Technologies

• Experience working with cloud services using one or more cloud providers such as Azure, GCP, or AWS.
• Experience with ETL tools such as Talend Big Data, Azure Data Factory, or similar solutions.

Soft Skills

• Proven ability to support and work with cross-functional teams in a dynamic environment.