Data Engineer with GCP

XPT Software Australia Pty Ltd; Ryde; On-site; AUD 110,000 - 150,000; Full time

Job summary

XPT Software Australia Pty Ltd is seeking a data engineering professional to design, develop, and maintain scalable data pipelines on Google Cloud Platform. You will implement ETL/ELT for batch and real-time processing using PySpark, Apache Beam, Dataflow, and Pub/Sub, and work with BigQuery for warehousing and SQL optimisation.

The role involves building data lakes and warehouses on Google Cloud, implementing data quality measures, and collaborating with cross-functional teams to deliver robust

Responsibilities

• Design, develop, and maintain scalable data pipelines on GCP.
• Build and optimise batch and real-time ETL/ELT workflows.
• Develop data processing solutions with Python, PySpark and Apache Beam.

Skills

GCP services

ETL/ELT

BigQuery

SQL development

Python

PySpark

Apache Beam

Cloud Dataflow

Pub/Sub

Cloud Composer

Job description

Key Responsibilities

Design, develop, and maintain scalable data pipelines using GCP services. Build and optimise ETL/ELT workflows for batch and real-time data processing. Work with Google BigQuery for data warehousing, SQL development, and query optimisation. Develop data processing solutions using Python, PySpark, and Apache Beam. Implement data integration using Cloud Dataflow, Cloud Composer, and Pub/Sub. Build and manage data lakes and data warehouses on Google Cloud. Implement data quali...