Data Engineer

V-ELEGANT PTE. LTD.; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

V-ELEGANT PTE. LTD. in Singapore is seeking a Data Engineer to create and maintain a single master record for each business entity, ensuring data accuracy and reliability.

You will build and optimize data pipelines across cloud, on-premises, and hybrid environments and collaborate with data scientists and analysts to turn data into actionable insights. The role requires experience with data lake/warehouse architectures, MD​M tooling, Spark and Kafka, SQL/NoSQL databases, and cloud platforms.

Qualifications

• 5+ years of experience as a data engineer.
• Familiar with data lake, data warehouse, and data lake house architectures.
• Experience with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality.
• Experience building and operating large-scale data lakes and data warehouses.
• Experience with Hadoop ecosystem and big data tools, including Spark and Kafka.
• Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee.
• Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage.
• Experience with stream-processing systems including Spark-Streaming.
• Experience working with Cloud services using one or more Cloud providers such as Azure, GCP, or AWS.
• Experience with Delta Lake and Databricks.
• Advanced working experience with relational SQL and NoSQL databases, including Hive, HBase, and Postgres.
• Deep understanding of SQL and the ability to optimize data queries.
• Experience with object-oriented/object function scripting languages: Python, Java, Scala, etc.
• A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
• Experience applying modern development principles (Scrum, TDD, continuous integration, and code reviews).
• Proven ability to support and work with cross-functional teams in a dynamic environment.
• Experience with ETL tools such as Talend Big Data, Azure Data Factory, etc.
• Experience working with Hortonworks Data Platform or Cloudera Data Platform.
• Experience with Metadata Management tools.
• Experience working on projects for Master Data Management.
• Exposure to Data Governance processes and tools.
• Proven ability in supporting and working with cross-functional teams in a dynamic environment.

Responsibilities

• Create and manage a single master record for each business entity, ensuring data consistency, accuracy, and reliability.
• Implement data governance processes, including data quality management, data profiling, data remediation, and automated data lineage.
• Create and maintain multiple robust and high-performance data processing pipelines within Cloud, Private Data Centre, and Hybrid data ecosystems.
• Assemble large, complex data sets from a wide variety of data sources.
• Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and Business users to derive actionable insights on data quality.
• Design and implement internal processes to automate manual workflows, optimize data delivery, and re-design infrastructure for greater scalability.
• Support and work with cross-functional teams in a dynamic environment.

Skills

Data engineering

Data governance

Cloud platforms

Spark

Kafka

SQL

NoSQL

MDM

Python

ETL

Tools

Informatica MDM

Talend Data Catalog

Semarchy xDM

IBM PIM IKC

Profisee

Talend Big Data

Azure Data Factory

Hadoop

Job description

Responsibilities

• Create and manage a single master record for each business entity,ensuring data consistency, accuracy, and reliability.
• Implement data governance processes, including data quality management,data profiling, data remediation, and automated data lineage.
• Create and maintain multiple robust and high-performance data processingpipelines within Cloud, Private Data Centre, and Hybrid data ecosystems.
• Assemble large, complex data sets from a wide variety of data sources.
• Collaborate with Data Scientists, Machine Learning Engineers, Business Analysts, and Business users to derive actionable insights on data quality.
• Design and implement internal processes to automate manual workflows,optimize data delivery, and re-design infrastructure for greater scalability.
• Support and work with cross-functional teams in a dynamic environment.

Requirements

• At least 5 years of experience ina data engineer role.
• Be familiar with data lake, data warehouse and data lake house architectures.
• Be familiar with MDM processes such as golden record creation, survivorship, reconciliation, enrichment, and quality.
• Experience building and operating large-scale data lakes and data ware houses.
• Experience with Hadoop ecosystem and big data tools, including Spark and Kafka.
• Experience with Master Data Management (MDM) tools and platforms such as Informatica MDM, Talend Data Catalog, Semarchy xDM, IBM PIM & IKC, or Profisee.
• Experience in data governance, including data quality management, data profiling, data remediation, and automated data lineage.
• Experience with stream-processing systems including Spark-Streaming.
• Experience working with Cloud services using one or more Cloud providerssuch as Azure, GCP, or AWS.
• Experience with Delta Lake and Databricks.
• Advanced working experience with relational SQL and NoSQL databases, including Hive, HBase, and Postgres.
• Deep understanding of SQL and the ability to optimize data queries.
• Experience with object-oriented/object function scripting languages:Python, Java, Scala, etc.
• A successful history of manipulating, processing, and extracting value from large, disconnected datasets.
• Experience applying modern development principles (Scrum, TDD,continuous integration, and code reviews).
• Proven ability to support and work with cross-functional teams in adynamic environment.
• Experience with ETL tools such as Talend Big Data, Azure Data Factory,etc.
• Experience working with Hortonworks Data Platform or Cloudera Data Platform.
• Experience with Metadata Management tools.
• Experience working on projects for Master Data Management.
• Exposure to Data Governance processes and tools.
• Proven ability in supporting and working with cross-functional teams in a dynamic environment.