Junior Data Engineer: Build Batch & Streaming Pipelines

Beyondsoft Singapore; Singapore; On-site; SGD 60,000 - 90,000; Full time

Job summary

Beyondsoft Singapore is hiring a Data Engineer to develop and maintain batch and streaming data pipelines, perform data mapping, transformation and validation, and monitor scheduled workflows.

You will deploy and maintain data workloads in containerised environments, troubleshoot issues, and collaborate with the team to uphold data quality and CI/CD practices. Candidates should have 2+ years in data engineering or backend roles with SQL and Java/Scala, plus Spark, Flink or Kafka experience.

Qualifications

• 2+ years of experience in data engineering or backend development.
• Familiarity with Agile development methodologies.
• Working knowledge of SQL and Java or Scala.
• Basic experience with Apache Spark, Apache Flink or Apache Kafka.
• Familiarity with Kubernetes, Docker and S3-compatible object storage.
• Experience with data mapping, transformation, validation and scheduled workflows.
• Familiarity with the SAFe framework.
• Exposure to MinIO or equivalent S3-compatible storage solutions.
• Familiarity with relational databases, Git, CI/CD and automated testing.
• Exposure to Apache Airflow or equivalent workflow orchestration tools.

Responsibilities

• Develop and maintain batch and streaming data pipelines.
• Perform data mapping, transformation and validation.
• Support scheduled data workflows and pipeline monitoring.
• Assist in deploying and maintaining data workloads in containerised environments.
• Troubleshoot data pipeline issues and support defect resolution.
• Work with the team to maintain data quality and support CI/CD processes.

Skills

SQL

Java or Scala

Airflow

Kubernetes

Docker

CI/CD

Data pipelines

Tools

Spark

Flink

Kafka

MinIO

S3-compatible storage

Job description

Beyondsoft Singapore is hiring a Data Engineer to develop and maintain batch and streaming data pipelines, perform data mapping, transformation and validation, and monitor scheduled workflows.

You will deploy and maintain data workloads in containerised environments, troubleshoot issues, and collaborate with the team to uphold data quality and CI/CD practices. Candidates should have 2+ years in data engineering or backend roles with SQL and Java/Scala, plus Spark, Flink or Kafka experience.