Senior Data Engineer: Real-Time Pipelines & Kubernetes

HYPERSCAL SOLUTIONS PTE. LTD.; Dublin; On-site; EUR 90,000 - 130,000; Full time

Job summary

Beyondsoft Consulting (Singapore) Pte. Ltd. is seeking a seasoned Data Engineer to design, build and operate large-scale batch and streaming data pipelines.

You will optimise distributed processing, manage data storage and ensure high availability in Kubernetes-based environments. The role requires 5+ years of data engineering experience with Spark, Flink and Kafka, plus proficiency in Java/Scala and SQL. Familiarity with CI/CD, GAo and S3-compatible storage is expected.

Qualifications

• Minimum 5 years of relevant data engineering experience.
• Hands-on experience with Apache Spark, Apache Flink and Apache Kafka.
• Proficiency in Java or Scala and SQL.
• Experience with Kubernetes and S3-compatible object storage.
• Knowledge of data quality controls, schema management, data lineage, checkpointing and recovery.
• Familiarity with GitLab and CI/CD practices.
• Experience with MinIO or equivalent S3-compatible storage solutions.
• Knowledge of Bronze/Silver/Gold data architecture and data integrity controls.
• Experience in high availability, backup, disaster recovery and inter-site synchronisation.
• Experience with Apache Airflow or equivalent workflow orchestration tools.
• Familiarity with SAFe and safety-critical or mission-critical systems.

Responsibilities

• Design, develop and deploy large-scale batch and real-time data processing pipelines.
• Build and optimise distributed data processing solutions.
• Deploy, manage and scale data workloads in Kubernetes environments.
• Manage data storage, quality, schema, metadata and lineage.
• Implement data checkpointing, recovery, backup and disaster recovery solutions.
• Ensure data integrity, retention and high availability across distributed storage systems.
• Maintain CI/CD pipelines and support automated deployment processes.

Skills

Data engineering

Apache Spark

Apache Flink

Apache Kafka

Java/Scala

SQL

Kubernetes

CI/CD

GitLab

Airflow

Data quality

Tools

MinIO

S3-compatible storage

Job description

Beyondsoft Consulting (Singapore) Pte. Ltd. is seeking a seasoned Data Engineer to design, build and operate large-scale batch and streaming data pipelines.

You will optimise distributed processing, manage data storage and ensure high availability in Kubernetes-based environments. The role requires 5+ years of data engineering experience with Spark, Flink and Kafka, plus proficiency in Java/Scala and SQL. Familiarity with CI/CD, GAo and S3-compatible storage is expected.