Senior Data Engineer: Real-Time Pipelines, K8s & Quality

Beyondsoft International (Singapore) Pte Ltd; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Beyondsoft International (Singapore) Pte. Ltd. invites a senior Data Engineer to design, develop and deploy large-scale batch and real-time data processing pipelines.

You will build distributed solutions, manage data storage, ensure data quality and lineage, and scale workloads in Kubernetes environments. You will implement recovery strategies, maintain CI/CD pipelines, and support automated deployments using tools like Apache Airflow, Spark, Flink and Kafka.

Qualifications

• 5+ years of data engineering experience.
• Hands-on experience with Apache Spark, Apache Flink and Apache Kafka.
• Proficiency in Java or Scala and SQL.
• Experience with Kubernetes and S3-compatible object storage.
• Knowledge of data quality controls, schema management, data lineage, checkpointing and recovery.
• Familiarity with GitLab and CI/CD practices.
• Experience with MinIO or equivalent S3-compatible storage solutions.
• Knowledge of Bronze/Silver/Gold data architecture and data integrity controls.
• Experience in high availability, backup, disaster recovery and inter-site synchronisation.
• Experience with Apache Airflow or equivalent workflow orchestration tools.
• Familiarity with SAFe and safety-critical or mission-critical systems.

Responsibilities

• Design, develop and deploy large-scale batch and real-time data processing pipelines.
• Build and optimise distributed data processing solutions.
• Deploy, manage and scale data workloads in Kubernetes environments.
• Manage data storage, quality, schema, metadata and lineage.
• Implement data checkpointing, recovery, backup and disaster recovery solutions.
• Ensure data integrity, retention and high availability across distributed storage systems.
• Maintain CI/CD pipelines and support automated deployment processes.

Skills

Spark

Flink

Kafka

Java

Scala

SQL

Airflow

Data quality

Data lineage

CI/CD

SAFe

Safety-critical

Tools

Kubernetes

MinIO

GitLab

S3-compatible storage

Job description

Beyondsoft International (Singapore) Pte. Ltd. invites a senior Data Engineer to design, develop and deploy large-scale batch and real-time data processing pipelines.

You will build distributed solutions, manage data storage, ensure data quality and lineage, and scale workloads in Kubernetes environments. You will implement recovery strategies, maintain CI/CD pipelines, and support automated deployments using tools like Apache Airflow, Spark, Flink and Kafka.