Senior Spark & Scala Data Engineer - Real-Time Pipelines

Peoplebank; Melbourne; On-site; AUD 110,000 - 165,000; Part time

Job summary

Peoplebank is seeking an experienced Spark & Scala Data Engineer to join an enterprise environment supporting critical infrastructure. The role covers multiple Australian cities and is onsite 1 day per fortnight with rolling 3-month contracts.

You will design scalable data pipelines with Spark and Scala, build real-time and batch processing solutions, and mentor junior engineers while collaborating with stakeholders to deliver projects on time and within budget.

Qualifications

• Extensive experience in Data Engineering or Big Data development.
• Hands-on Scala, Java and Apache Spark.
• Experience building real-time and batch data pipelines.
• Strong knowledge of Kafka, Hive, HBase, HDFS and the Hadoop ecosystem.
• Strong SQL and data analysis skills.
• Experience with CI/CD, automated testing and Agile delivery.
• Strong communication and stakeholder management skills.

Responsibilities

• Design and develop scalable data pipelines using Spark and Scala.
• Build and support batch and real-time data processing solutions.
• Analyse requirements and provide fit-for-purpose technical designs.
• Develop APIs, data services and reusable engineering frameworks.
• Deliver high-quality, well-tested code and perform code reviews.
• Support CI/CD, automation and production operations.
• Collaborate with stakeholders to deliver projects on time and within budget.
• Mentor junior engineers and contribute to continuous improvement.

Skills

Data Engineering

Scala

Java

Apache Spark

Kafka

Hive

HBase

HDFS

SQL

CI/CD & Agile

Stakeholder management

Tools

Kafka

Hive

HBase

HDFS

CI/CD tools

Job description

Peoplebank is seeking an experienced Spark & Scala Data Engineer to join an enterprise environment supporting critical infrastructure. The role covers multiple Australian cities and is onsite 1 day per fortnight with rolling 3-month contracts.

You will design scalable data pipelines with Spark and Scala, build real-time and batch processing solutions, and mentor junior engineers while collaborating with stakeholders to deliver projects on time and within budget.