Spark & Scala Data Engineer — Real-Time Pipelines

Peoplebank; City of Brisbane; On-site; AUD 120,000 - 180,000; Full time

Job summary

Peoplebank is seeking an experienced Senior Process Specialist to join our enterprise team supporting critical infrastructure in Melbourne, VIC. The role is a rolling 3x month contract and requires 1 day per fortnight onsite, with opportunities to influence data platforms and delivery pipelines.

You will design scalable data pipelines, develop batch and real-time solutions, mentor engineers, and collaborate with stakeholders to deliver projects on time and within budget.

Qualifications

• Extensive experience in Data Engineering or Big Data development.
• Strong hands-on experience with Scala, Java and Apache Spark.
• Experience building real-time and batch data pipelines.
• Strong knowledge of Kafka, Hive, HBase, HDFS and the Hadoop ecosystem.
• Experience designing and supporting large-scale Big Data platforms.
• Strong SQL and data analysis skills.
• Experience with CI/CD, automated testing and Agile delivery.
• Strong communication and stakeholder management skills.

Responsibilities

• Design and develop scalable data pipelines using Spark and Scala.
• Build and support batch and real-time data processing solutions.
• Analyse requirements and provide fit-for-purpose technical designs.
• Develop APIs, data services and reusable engineering frameworks.
• Deliver high-quality, well-tested code and perform code reviews.
• Support CI/CD, automation and production operations.
• Collaborate with stakeholders to deliver projects on time and within budget.
• Mentor junior engineers and contribute to continuous improvement.

Skills

Data Engineering

Big Data

Scala

Java

Apache Spark

Kafka

Hive

HBase

HDFS

Hadoop

SQL

CI/CD

Agile Delivery

Stakeholder Management

Communication

Tools

Hive

HBase

HDFS

Apache Hadoop

Airflow

Job description

Peoplebank is seeking an experienced Senior Process Specialist to join our enterprise team supporting critical infrastructure in Melbourne, VIC. The role is a rolling 3x month contract and requires 1 day per fortnight onsite, with opportunities to influence data platforms and delivery pipelines.

You will design scalable data pipelines, develop batch and real-time solutions, mentor engineers, and collaborate with stakeholders to deliver projects on time and within budget.