GCP Data Engineer: Real-Time Pipelines & BigQuery Expert

XPT Software Australia Pty Ltd; Sydney; Presencial; AUD 90.000 - 120.000; Jornada completa

Descripción de la vacante

XPT Software Australia Pty Ltd in Sydney is seeking a Data Engineer with GCP expertise to design and implement scalable data pipelines. You will work on BigQuery data warehousing, SQL development, and real-time processing using Python and Apache Beam.

The role emphasizes data quality, security, and governance while collaborating with analysts and business teams. The contract position offers exposure to large-scale cloud data architectures, cost-efficient pipeline design, and close interaction

Formación

• Experience designing and implementing scalable data pipelines on Google Cloud Platform (GCP).
• Proficient in BigQuery data warehousing, SQL development, and query optimization.
• Strong programming skills in Python and SQL for data processing tasks.
• Experience with batch and real-time data processing architectures and ETL/ELT pipelines.

Responsabilidades

• Design, develop, and maintain scalable data pipelines using GCP services.
• Build and optimise ETL/ELT workflows for batch and real-time data processing.
• Work with Google BigQuery for data warehousing, SQL development, and query optimisation.
• Develop data processing solutions using Python, PySpark, and Apache Beam.
• Implement data integration using Cloud Dataflow, Cloud Composer, and Pub/Sub.
• Build and manage data lakes and data warehouses on Google Cloud.
• Implement data quality checks, data validation, and monitoring.
• Optimise pipeline performance, scalability, and cost efficiency.
• Collaborate with data analysts, data scientists, architects, and business teams.
• Ensure data security, governance, and compliance best practices.

Conocimientos

Google Cloud Platform

BigQuery

Python

SQL

Cloud Dataflow

Apache Beam

Cloud Storage

Herramientas

Cloud Dataflow

Apache Beam

Cloud Composer

Pub/Sub

GCS

Descripción del empleo

XPT Software Australia Pty Ltd in Sydney is seeking a Data Engineer with GCP expertise to design and implement scalable data pipelines. You will work on BigQuery data warehousing, SQL development, and real-time processing using Python and Apache Beam.

The role emphasizes data quality, security, and governance while collaborating with analysts and business teams. The contract position offers exposure to large-scale cloud data architectures, cost-efficient pipeline design, and close interaction