Senior Data Engineer: Real-Time Pipelines on Kubernetes

Beyondsoft Singapore; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Beyondsoft Singapore is seeking an experienced Data Engineer to design, build, and deploy large-scale batch and real-time data processing pipelines across distributed systems. You will optimise data workloads in Kubernetes, manage storage quality and lineage, and implement robust checkpointing and disaster recovery strategies.

The role requires hands-on Spark, Flink and Kafka experience, proficiency in Java/Scala and SQL, and familiarity with SAFe and CI/CD practices.

Qualifications

• 5+ years of data engineering experience.
• Hands-on with Apache Spark, Flink and Kafka.
• Proficiency in Java or Scala and SQL.
• Experience with Kubernetes and S3-compatible object storage.
• Knowledge of data quality controls, schema management, data lineage, checkpointing and recovery.
• Familiarity with GitLab and CI/CD practices.
• Experience with MinIO or equivalent S3-compatible storage solutions.
• Knowledge of Bronze/Silver/Gold data architecture and data integrity controls.
• Experience in high availability, backup, disaster recovery and inter-site synchronisation.
• Experience with Apache Airflow or equivalent workflow orchestration tools.
• Familiarity with SAFe and safety-critical or mission-critical systems.

Responsibilities

• Design, develop and deploy large-scale batch and real-time data processing pipelines.
• Build and optimise distributed data processing solutions.
• Deploy, manage and scale data workloads in Kubernetes environments.
• Manage data storage, quality, schema, metadata and lineage.
• Implement data checkpointing, recovery, backup and disaster recovery solutions.
• Ensure data integrity, retention and high availability across distributed storage systems.
• Maintain CI/CD pipelines and support automated deployment processes.

Skills

5+ years data engineering

Data quality

Data lineage

Disaster recovery

SAFe

Safety-critical systems

Data replication

Tools

Apache Spark

Apache Flink

Apache Kafka

Kubernetes

GitLab

Apache Airflow

MinIO

Job description

Beyondsoft Singapore is seeking an experienced Data Engineer to design, build, and deploy large-scale batch and real-time data processing pipelines across distributed systems. You will optimise data workloads in Kubernetes, manage storage quality and lineage, and implement robust checkpointing and disaster recovery strategies.

The role requires hands-on Spark, Flink and Kafka experience, proficiency in Java/Scala and SQL, and familiarity with SAFe and CI/CD practices.