Senior Data Engineer: Real-Time Pipelines & Spark

hyperscal solutions pte. ltd.; Singapore; On-site; SGD 90,000 - 140,000; Full time

Job summary

Beyondsoft Consulting (Singapore) Pte. Ltd in Singapore is seeking an experienced Data Engineer to design, build and operate large-scale batch and real-time data pipelines.

You will optimise distributed data processing, deploy and scale workloads in Kubernetes, and ensure data quality and availability across storage systems. The role requires 5+ years of data engineering experience with Spark, Flink and Kafka, plus Java/Scala and SQL; familiarity with MinIO or S3-compatible storage, GitLab

Qualifications

• Minimum 5 years of relevant data engineering experience.
• Hands-on experience with Apache Spark, Apache Flink and Apache Kafka.
• Proficiency in Java or Scala and SQL.
• Experience with Kubernetes and S3-compatible object storage.
• Knowledge of data quality controls, schema management, data lineage, checkpointing and recovery.
• Familiarity with GitLab and CI/CD practices.
• Experience with MinIO or equivalent S3-compatible storage solutions.
• Knowledge of Bronze/Silver/Gold data architecture and data integrity controls.
• Experience in high availability, backup, disaster recovery and inter-site synchronisation.
• Experience with Apache Airflow or equivalent workflow orchestration tools.
• Familiarity with SAFe and safety-critical or mission-critical systems.

Responsibilities

• Design, develop and deploy large-scale batch and real-time data processing pipelines.
• Build and optimise distributed data processing solutions.
• Deploy, manage and scale data workloads in Kubernetes environments.
• Manage data storage, quality, schema, metadata and lineage.
• Implement data checkpointing, recovery, backup and disaster recovery solutions.
• Ensure data integrity, retention and high availability across distributed storage systems.
• Maintain CI/CD pipelines and support automated deployment processes.

Skills

5+ years data engineering

Spark

Flink

Kafka

Java/Scala

SQL

Data quality controls

Data lineage

Checkpointing & recovery

SAFe

Safety-critical systems

Tools

Kubernetes

S3-compatible storage

MinIO

GitLab CI/CD

Airflow

Data architecture (Bronze/Silver/Gold)

Job description

Beyondsoft Consulting (Singapore) Pte. Ltd in Singapore is seeking an experienced Data Engineer to design, build and operate large-scale batch and real-time data pipelines.

You will optimise distributed data processing, deploy and scale workloads in Kubernetes, and ensure data quality and availability across storage systems. The role requires 5+ years of data engineering experience with Spark, Flink and Kafka, plus Java/Scala and SQL; familiarity with MinIO or S3-compatible storage, GitLab