GCP Data Engineer: Real-Time Pipelines & Generative AI

FR Consultancy; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

FR Consultancy in Sydney seeks a hands-on Data Platform Architect with deep GCP expertise to design and implement large-scale data platforms. You will lead batch and real-time data pipelines and governance initiatives across on-prem and cloud sources.

The role emphasizes building scalable, transaction-focused architectures, applying Generative AI integrations, and using tools like GCP, BigQuery, Spanner, Dataplex, Data Catalog, Cloud Composer, Terraform, and Confluent Kafka.

Qualifications

• Hands-on Python and Java development.
• Experience with GCP data and analytics.
• Experience designing and implementing data pipelines and governance.

Responsibilities

• In-depth knowledge and experience of GCP data and analytics technologies.
• Designing Large scale Transaction Data Platform architecture for migrating data platforms to GCP cloud-native services.
• Define solution architecture and detailed design, and perform hands-on implementation for data pipelines for batch and event-driven ingestion and processing from diverse sources.
• Real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery.
• Automatic transpilation of legacy code (HIVE, Teradata, Python logic) to BigQuery SQL.
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform.
• Orchestration setup and Cloud Composer DAG development for data pipeline workflows.
• Data governance using GCP tooling (Dataplex, Data Catalog) with Generative AI integrations in enterprise environments.

Skills

Python

Java

Tools

GCP

BigQuery

Spanner

Dataplex

Data Catalog

Cloud Composer

Terraform

Confluent Kafka

Job description

FR Consultancy in Sydney seeks a hands-on Data Platform Architect with deep GCP expertise to design and implement large-scale data platforms. You will lead batch and real-time data pipelines and governance initiatives across on-prem and cloud sources.

The role emphasizes building scalable, transaction-focused architectures, applying Generative AI integrations, and using tools like GCP, BigQuery, Spanner, Dataplex, Data Catalog, Cloud Composer, Terraform, and Confluent Kafka.