Senior Data Engineer — Real-Time Pipelines & Kubernetes

HYPERSCAL SOLUTIONS PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Beyondsoft Consulting (Singapore) Pte. Ltd is seeking an experienced Data Engineer to design and implement large-scale data pipelines for batch and real-time workloads.

You will work with Spark, Flink, Kafka, Java/Scala and SQL, operating in Kubernetes environments and ensuring data quality and reliability. The role requires hands-on experience with S3-compatible storage and CI/CD practices, with a focus on high availability and disaster recovery.

Qualifications

• Minimum 5 years of data engineering experience.
• Hands-on with Apache Spark, Apache Flink and Apache Kafka.
• Proficiency in Java or Scala and SQL.
• Experience with Kubernetes and S3-compatible object storage.
• Knowledge of data quality controls, schema management, data lineage and recovery.
• Familiarity with GitLab CI/CD and SAFe.

Responsibilities

• Design, develop and deploy large-scale batch and real-time data processing pipelines.
• Build and optimise distributed data processing solutions.
• Deploy, manage and scale data workloads in Kubernetes environments.
• Manage data storage, quality, schema, metadata and lineage.
• Implement data checkpointing, recovery, backup and disaster recovery solutions.
• Ensure data integrity, retention and high availability across distributed storage systems.
• Maintain CI/CD pipelines and support automated deployment processes.

Skills

Apache Spark

Apache Flink

Apache Kafka

Java/Scala

SQL

Airflow

SAFe

GitLab CI/CD

MinIO or S3 storage

Tools

Kubernetes

S3-compatible storage

Job description

Beyondsoft Consulting (Singapore) Pte. Ltd is seeking an experienced Data Engineer to design and implement large-scale data pipelines for batch and real-time workloads.

You will work with Spark, Flink, Kafka, Java/Scala and SQL, operating in Kubernetes environments and ensuring data quality and reliability. The role requires hands-on experience with S3-compatible storage and CI/CD practices, with a focus on high availability and disaster recovery.