Data Engineer: Spark, Kafka & Big Data Pipelines

Mindrift; Canberra; On-site; AUD 110,000 - 140,000; Full time

Job summary

Nuvento in Canberra seeks a Data Engineer to build and maintain Spark pipelines, ingest data from Kafka, and ensure robust data processing for a Federal Government client. You will collaborate with cross-functional teams to deliver scalable data solutions and maintain data quality across the data lake.

The role emphasizes hands-on development, troubleshooting, and staying current with big data technologies such as Spark, Kafka, and Airflow.

Qualifications

• Bachelor's or master's in Computer Science, Software Engineering, or related field.
• 3+ years of experience in data engineering or software development.
• Strong programming skills in Scala, Python, or Java.
• Expertise in Spark, Kafka, and big data processing concepts.
• Experience with data storage solutions (HDFS or S3) and data serialization formats.
• Knowledge of database concepts, data modelling, schema design, and normalisation.
• Familiarity with monitoring/logging tools and workflow management tools (e.g., Airflow).
• Experience with version control systems (Git).

Responsibilities

• Maintain and develop Spark pipelines to process large-scale data sets.
• Ingest tables into data lake from Kafka using batch, micro-batch, or real-time processing.
• Run Spark jobs for ingestion, curation, and extraction of data.
• Alter schemas and configs to meet business requirements and ensure data quality.
• Attend meetings regarding data ingestion and collaborate with cross-functional teams.
• Troubleshoot and resolve bugs in data processing pipelines.
• Monitor performance and troubleshoot using logging/monitoring tools.
• Document best practices and develop standards for data pipelines.
• Stay up-to-date with emerging technologies and trends in big data processing.

Skills

Spark

Kafka

Scala

Python

Java

Education

Bachelor's or Master's in CS/SE

Tools

HDFS

S3

Airflow

Git

Job description

Nuvento in Canberra seeks a Data Engineer to build and maintain Spark pipelines, ingest data from Kafka, and ensure robust data processing for a Federal Government client. You will collaborate with cross-functional teams to deliver scalable data solutions and maintain data quality across the data lake.

The role emphasizes hands-on development, troubleshooting, and staying current with big data technologies such as Spark, Kafka, and Airflow.