Data Engineer

Hydrogen Group; Sydney; On-site; AUD 166,000 - 240,000; Full time

Job summary

Hydrogen Group in Australia is seeking a Data Engineer for a leading financial organisation. You will work on cutting-edge ETL design, Big Data tooling, and real-time to batch data ingestion using GCP, with exposure to Azure and Databricks. You will build reusable ingestion code, populate Data Lake and data models, design ETL pipelines, and deliver data curation jobs with CI/CD. Strong SQL, Spark, Kafka, and cloud experience are essential for success.

Qualifications

• Experience with cloud platforms Azure and GCP.
• Strong Python and Databricks development skills.
• Hands-on with Spark, Kafka, and SQL/PL-SQL.

Responsibilities

• Create repeatable ingestion and curation code per architecture.
• Populate Data Lake and models for new sources.
• Design ETL processes and pipelines where required.
• Deliver data curation jobs with development, testing, and CI/CD.
• Ingest structured and unstructured data into GCP in real-time or batch.
• Ensure production services are configured to protect data.

Skills

Azure

GCP

Python

Databricks

Spark

Kafka

SQL

PL/SQL

Low-level design

Reusable framework

Tools

Snowflake

Job description

The Client My client is a leading financial organisation who are continuing their journey with multiple projects where they are looking for Data Engineers. This client is working on multiple projects which would give you an exciting opportunity to cutting edge Technology tools.

The Role This role offers a great chance to join a team of Engineering Manager's, Data experts and Project Manager’s working on cutting edge, highly complex ETL design and Big Data tools. The position will also give you the chance to gain exposure to a range of other languages/technologies.

The ideal candidate will have experience working on similar projects working heavily with Big Data Tools and ETL design. The key responsibilities will include.

• Create repeatable and reusable ingestion and curation code in accordance with the architecture
• Populate Data Lake and Data model for new the sources that haven’t been incorporated into big data ecosystem
• Low level design implementing ETL processes/pipelines where required
• Delivery of data curation Jobs, including development, testing and implementation using CI/CD framework
• Ingest structured / unstructured data in Real time, Near real time or batch cadence into GCP
• Responsible for ensuring the production services are optimally configured and aligned to best practices to protect production data.

Candidate Profile Looking for the following skills/experience.

• Working experience with cloud platforms such as Azure & GCP
• Strong experience with Python & Databricks
• Proven experience building data working with tools such as Spark and Kafka
• Highly skilled in all aspects of SQL and PL/SQL
• Low level design or reusable framework experience highly regarded
• Working experince with tools such Snowflake or Databricks

What’s on offer? This is an excellent opportunity to work on a great contract where you will have access to market leading tools, expert colleagues and a modern working environment. This opportunity offers a competitive daily rate as well as other excellent benefits.