Data Engineer / Consultant

WSH Experts Pte Ltd; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A leading data solutions company in Singapore is seeking a skilled data engineer to manage data governance processes and build large-scale data lakes. The role involves collaborating with cross-functional teams to derive actionable insights and optimizing data workflows. Candidates should possess strong experience with Cloudera, big data tools, and cloud services. This position offers an exciting opportunity to work in a dynamic environment with innovative data solutions.

Qualifications

• Building and operating large-scale data lakes and data warehouses.
• Experience with Cloudera Hadoop ecosystem and big data tools.
• Familiarity with Master Data Management processes.

Responsibilities

• Create and manage master records ensuring data consistency.
• Implement data governance processes effectively.
• Collaborate with cross-functional teams to derive insights.

Skills

Building large-scale data lakes

Cloudera Hadoop ecosystem

Master Data Management (MDM)

Data governance

Stream-processing systems

Cloud services (Azure, GCP, AWS)

SQL and NoSQL databases

Tools

Informatica MDM

Talend Data Catalog

Databricks

Spark

Kafka

Delta Lake

Job description

Job Description

• Create and manage a single master record for each business entity, ensuring data consistency, accuracy, and reliability.
• Implement data governance processes, including data quality management, data profiling, data remediation, and automated data lineage.
• Create and maintain multiple robust and high-performance data processing pipelines within Cloud, Private Data Centre, and Hybrid data ecosystems.
• Assemble large, complex data sets from a wide variety of data sources.
• Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and Business users to derive actionable insights and reliable foresights into customer acquisition, operational efficiency, and other key business performance metrics.
• Develop, deploy, and maintain multiple microservices, REST APIs, and reporting services.
• Design and implement internal processes to automate manual workflows, optimize data delivery, and re-design infrastructure for greater scalability.
• Establish expertise in designing, analyzing, and troubleshooting large-scale distributed systems.
• Support and work with cross-functional teams in a dynamic environment.

Job Requirements

• Experience building and operating large-scale data lakes and data warehouses.
• Experience with Cloudera Hadoop ecosystem, Impala & Trino as well as big data tools, including Spark and Kafka.
• Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee.
• Familiarity with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality.
• Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage.
• Experience with stream-processing systems including Spark-Streaming.
• Experience working with Cloud services using one or more Cloud providers such as Azure, GCP, or AWS.
• Experience with Delta Lake and Databricks.
• Advanced working experience with relational SQL and NoSQL databases, including Hive, HBase, and Postgres.
• Deep understanding of SQL and the ability to optimize data queries.