Data Engineer

ADECCO PERSONNEL PTE LTD; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

ADECCO PERSONNEL PTE LTD is seeking a Data Engineer to design and build scalable ETL/ELT data pipelines using Python, PySpark and Spark. You will manage lakehouse architectures with Iceberg tables and implement metadata and partitioning strategies.

Responsibilities include deploying pipelines with Kubernetes, Docker, CI/CD, and DevOps practices in on-prem environments, providing L3 support, and ensuring data security and governance while collaborating with stakeholders.

Qualifications

• Demonstrated experience in data engineering/pipeline development with Python and SQL Server.

Responsibilities

• Design and develop scalable ETL/ELT data pipelines using Python, PySpark, MSBI and Apache Spark.
• Build and manage lake house/data platforms, including data modelling, partitioning, metadata and Iceberg tables.
• Deploy and manage data pipelines using Kubernetes, Docker, CI/CD and DevOps practices in on-premise environments.
• Provide L3 support, optimise data platform performance, ensure security/governance and collaborate with technical and business stakeholders.

Skills

Python

SQL

MSBI

Tools

Kubernetes

Docker

Apache Spark/PySpark

Apache Airflow

Iceberg

MongoDB

Cassandra

Git

IaC

Job description

Role & Responsibilities

Responsibilities

• Design and develop scalable ETL/ELT data pipelines using Python, PySpark, MSBI and Apache Spark.
• Build and manage lake house/data platforms, including data modelling, partitioning, metadata and Iceberg tables.
• Deploy and manage data pipelines using Kubernetes, Docker, CI/CD and DevOps practices in on-premise environments.
• Provide L3 support, optimise data platform performance, ensure security/governance and collaborate with technical and business stakeholders.

Requirements

• Demonstrated experience in data engineering/pipeline development, with strong Python, SQL Server and MSBI expertise.
• Hands-on experience with Kubernetes, Docker, Apache Spark/PySpark and Apache Airflow for data processing and orchestration.
• Strong data modelling skills across relational and NoSQL databases, including MongoDB or Cassandra.
• Experience with lakehouse/data lake architecture, Iceberg, partitioning, metadata management, CI/CD, Git and IaC.
• Strong understanding of data security, governance, RBAC and data masking.

NOTE: - Only shortlisted candidates will be contacted back.