Junior Data Engineer: Scalable Batch & Streaming Pipelines

HYPERSCAL SOLUTIONS PTE. LTD.; Dublin; On-site; EUR 65,000 - 90,000; Full time

Job summary

Beyondsoft International offers a data engineering role focused on building and maintaining batch and streaming data pipelines. You will map, transform and validate data, support scheduled workflows and monitor pipelines, and help deploy workloads in containerised environments.

Join a team that values data quality, Agile practices and modern tech like Spark, Kafka and Airflow to enable scalable data solutions across markets.

Qualifications

• Minimum 2 years of experience in data engineering or backend development.
• Familiarity with Agile development methodologies.
• Working knowledge of SQL and Java or Scala.
• Basic experience with Apache Spark, Apache Flink or Apache Kafka.
• Familiarity with Kubernetes, Docker and S3-compatible object storage.
• Experience with data mapping, transformation, validation and scheduled workflows.
• Familiarity with the SAFe framework.
• Exposure to MinIO or equivalent S3-compatible storage solutions.
• Familiarity with relational databases, Git, CI/CD and automated testing.
• Exposure to Apache Airflow or equivalent workflow orchestration tools.
• Knowledge of batch and stream-processing technologies.

Responsibilities

• Develop and maintain batch and streaming data pipelines.
• Perform data mapping, transformation and validation.
• Support scheduled data workflows and pipeline monitoring.
• Assist in deploying and maintaining data workloads in containerised environments.
• Troubleshoot data pipeline issues and support defect resolution.
• Work with the team to maintain data quality and support CI/CD processes.

Skills

Batch processing

Stream processing

SQL

Java/Scala

Apache Spark

Apache Kafka

Apache Flink

Kubernetes

Docker

S3 storage

MinIO

Git

CI/CD

Airflow

Agile

Relational DBs

Education

Tools

Kubernetes

Docker

S3-compatible storage

MinIO

Git

CI/CD

Apache Airflow

Job description

Beyondsoft International offers a data engineering role focused on building and maintaining batch and streaming data pipelines. You will map, transform and validate data, support scheduled workflows and monitor pipelines, and help deploy workloads in containerised environments.

Join a team that values data quality, Agile practices and modern tech like Spark, Kafka and Airflow to enable scalable data solutions across markets.