Data Engineer

ZENTACT SYSTEMS PTE. LTD.; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Zentact Systems PTE. LTD. is seeking a Data Engineer in Singapore to migrate data pipelines to the GDP data acquisition framework and build robust ingestion scripts for the data lake.

You will develop and manage end-to-end ETL/ELT workflows with strong emphasis on data quality and reliability, while deploying containerized apps with Docker and Kubernetes. The role requires hands-on experience with modern data lake technologies, real-time streaming with Kafka, and workflow orchestration via

Qualifications

• Strong proficiency in Linux, Python, and Shell scripting.
• Hands-on experience with Docker, Kubernetes, and container orchestration.
• Experience with MinIO and Azure Data Lake Storage using S3 protocols.
• Experience with Apache Iceberg, Kafka, Airflow, DataHub, Trino, and Ranger.

Responsibilities

• Migrate data pipelines to GDP data acquisition framework.
• Develop and deliver data ingestion scripts for loading data into the data lake.
• Develop and manage ETL/ELT workflows with data quality, integrity and reliability.
• Automate data quality checks and validation within data pipelines.
• Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
• Work with modern data lake/warehouse technologies such as Iceberg.
• Implement real-time data streaming solutions using Kafka.
• Orchestrate complex workflows using Airflow.
• Integrate with data catalog and governance tools such as DataHub and Ranger.
• Collaborate with cross-functional teams to understand business requirements and deliver data solutions.

Skills

Linux

Python

Shell scripting

Tools

Docker

Kubernetes

Airflow

Kafka

Apache Iceberg

Datahub

Ranger

Trino

MinIO

Azure Data Lake Storage

Job description

Position : Data Engineer

• Migratedatapipelines from existingdataacquisition framework to the new GDPdataacquisition framework
• Configure,developand deliverdataingestion scripts for loadingdatainto T1datalayer
• Develop and manage ETL/ELT workflows, ensuringdataquality, integrity, and reliability.
• Integrate and automatedataquality checks and validation processes withindatapipelines.
• Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes.
• Work with moderndatalake and warehouse technologies such as Iceberg
• Implement real-timedatastreaming solutions using Kafka.
• Orchestrate complex workflows using Airflow.
• Integrate withdatacatalogand governance tools such asDatahub and Ranger.
• Collaborate with cross-functional teams to understand business requirements and deliverdatasolutions.
• Ensure security, compliance, and best practices indatamanagement and governance.
• Strongproficiencyin Linux, Python, and Shell scripting.
• Hands-on experience with Docker, Kubernetes, and container orchestration.
• Hands-on experience with Minio and AzureDataLake Storage (ADLS) using S3 protocols.
• Experience withApacheIceberg, Kafka, Airflow,Datahub, Trino, and Ranger. Work Location : Singapore.