GCP Data Engineer

FR Consultancy; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

FR Consultancy in Sydney seeks a hands-on Data Platform Architect with deep GCP expertise to design and implement large-scale data platforms. You will lead batch and real-time data pipelines and governance initiatives across on-prem and cloud sources.

The role emphasizes building scalable, transaction-focused architectures, applying Generative AI integrations, and using tools like GCP, BigQuery, Spanner, Dataplex, Data Catalog, Cloud Composer, Terraform, and Confluent Kafka.

Qualifications

• Hands-on Python and Java development.
• Experience with GCP data and analytics.
• Experience designing and implementing data pipelines and governance.

Responsibilities

• In-depth knowledge and experience of GCP data and analytics technologies.
• Designing Large scale Transaction Data Platform architecture for migrating data platforms to GCP cloud-native services.
• Define solution architecture and detailed design, and perform hands-on implementation for data pipelines for batch and event-driven ingestion and processing from diverse sources.
• Real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery.
• Automatic transpilation of legacy code (HIVE, Teradata, Python logic) to BigQuery SQL.
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform.
• Orchestration setup and Cloud Composer DAG development for data pipeline workflows.
• Data governance using GCP tooling (Dataplex, Data Catalog) with Generative AI integrations in enterprise environments.

Skills

Python

Java

Tools

GCP

BigQuery

Spanner

Dataplex

Data Catalog

Cloud Composer

Terraform

Confluent Kafka

Job description

Status:

AU valid visa with full work rights

Must have:

• Programming knowledge and willingness to be hands-on - Python, Java

Key Responsibilities

• In-depth knowledge and experience of GCP data and analytics technologies
• Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
• Define solution architecture and detailed design, and perform hands-on implementation for data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.
• Real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
• Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
• Orchestration setup and Cloud Composer DAG development for data pipeline workflows
• Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog) Experience applying Generative AI technologies and integrations in enterprise environments.
• Tools and languages experience Required