21. Data Engineer (Senior) - KT

dci consultants private limited; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

dci consultants private limited is seeking a data engineer in Singapore with 5–8 years of data engineering experience to design and deploy large-scale data systems. The role concentrates on building batch and streaming pipelines and ensuring robust data quality and governance.

You will design pipelines using Spark, Flink and Kafka, program in Java or Scala with SQL, and deploy workloads in Kubernetes. Experience with S3-compatible storage, data-quality controls, schema/lineage, checkpointing,

Qualifications

• 5–8 years of data engineering experience.
• Strong hands-on experience designing batch and streaming pipelines using Apache Spark, Apache Flink and Apache Kafka.
• Proficiency in Java or Scala for distributed data processing, together with SQL.
• Experience deploying and scaling data workloads in containerised Kubernetes environments.
• Experience with S3-compatible object storage, data-quality controls, schema and lineage management, checkpointing and recovery, GitLab and CI/CD practices.

Skills

Data engineering

Spark

Flink

Kafka

Java/Scala

SQL

Kubernetes

S3 storage

Data quality

Schema lineage

Checkpointing

GitLab CI/CD

Job description

Mandatory Requirements

• 5–8 years of data engineering experience, including building and deploying large-scale data systems


• Strong hands-on experience designing batch and streaming pipelines using Apache Spark, Apache Flink and Apache Kafka


• Proficiency in Java or Scala for distributed data processing, together with SQL


• Experience deploying and scaling data workloads in containerised Kubernetes environments


• Experience with S3-compatible object storage, data-quality controls, schema and lineage management, checkpointing and recovery, GitLab and CI/CD practices

Preferred Requirements

• Experience working within the SAFe framework


• Experience developing data solutions for safety-critical and mission-critical systems


• Experience with MinIO or equivalent S3-compatible object storage; schema and lineage design; data-quality checks; metadata preservation; bronze/silver/gold layered processing; and data-integrity controls including erasure coding, checksums, bit-rot protection, background healing and strong consistency


• Experience implementing checkpointing and recovery across hot and cold reboots, retention and lifecycle controls for the contracted retention period, high availability, backup and recovery, and operational/disaster-recovery deployment with inter-site synchronisation

Equivalent Technologies

• Equivalent large-scale batch or stream-processing frameworks may be considered where the candidate demonstrates transferable distributed-data-system experience; Apache Airflow or an equivalent workflow-orchestration platform and other S3-compatible object stores remain relevant supporting technologies

**We regret to inform that only shortlisted candidates will be notified. Personal data collected will be used for recruitment purposes**