GCP Data Platform Engineer — Real-Time Pipelines & AI

CareCone Group; City of Melbourne; On-site; AUD 140,000 - 220,000; Full time

Job summary

CareCone Group seeks a senior data platform architect to lead the design and hands-on implementation of cloud-native data solutions on Google Cloud Platform. You will shape architecture and detailed designs, delivering scalable data pipelines for batch and real-time ingestion.

You will work with Confluent Kafka, GCS, Spanner and BigQuery, and drive CI/CD with Cloud Build and Terraform while enabling data governance with Dataplex and Data Catalog, including Generative AI integrations for

Qualifications

• Proficiency in Python and Java.
• Experience with GCP data and analytics technologies.
• Hands-on software development and cloud data engineering skills.

Responsibilities

• Design large-scale transaction data platform architecture for migrating open-source or public-cloud data platforms to GCP native services.
• Define solution architecture and detailed design, and perform hands-on implementation for data pipelines (batch and event-driven).
• Build data pipelines ingesting from on-prem files, on-prem databases, and APIs.
• Implement real-time ingestion and event processing from Confluent Kafka and GCS to Spanner and BigQuery.
• Create CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform.
• Set up orchestration and Cloud Composer DAG development for data pipelines.
• Apply data governance using Dataplex and Data Catalog.
• Explore Generative AI integrations in enterprise environments.

Skills

Python

Java

GCP data analytics

Tools

Cloud Build

Artifact Registry

Terraform

Cloud Composer

Dataplex

Data Catalog

Confluent Kafka

BigQuery

Spanner

GCS

Job description

CareCone Group seeks a senior data platform architect to lead the design and hands-on implementation of cloud-native data solutions on Google Cloud Platform. You will shape architecture and detailed designs, delivering scalable data pipelines for batch and real-time ingestion.

You will work with Confluent Kafka, GCS, Spanner and BigQuery, and drive CI/CD with Cloud Build and Terraform while enabling data governance with Dataplex and Data Catalog, including Generative AI integrations for