Senior GCP Data Architect for Real-Time & Batch Pipelines

Delivery Centric Pty Ltd; Sydney; On-site; AUD 180,000 - 250,000; Full time

Job summary

Delivery Centric Pty Ltd seeks a Senior GCP Data Architect and Data Engineering Lead to design and deliver large-scale real-time and batch data platforms on Google Cloud. You will combine solution architecture, hands-on data engineering, leadership and customer collaboration to drive complex data initiatives.

You will lead architecture, data modelling and pipelines, guiding teams on GCP-native services, Dataflow, BigQuery, Spanner and AI-enabled data solutions.

Qualifications

• Strong hands-on experience with Google Cloud Platform data and analytics technologies.
• Strong experience with GCP Dataflow, BigQuery and Cloud Spanner.
• Experience with real-time/event-driven data ingestion using Confluent Kafka and Pub/Sub.
• Experience with Cloud Composer and Cloud Workflows.
• Strong experience with Terraform and Cloud Build for data engineering workloads.
• Hands-on programming experience with Python and Java.
• Experience designing and implementing batch and event-driven data pipelines from sources including databases, files, APIs and streaming platforms.
• Experience with BigQuery performance optimisation.
• Experience with CI/CD, infrastructure automation and production data workload deployment.
• Strong understanding of data architecture, solution design and engineering best practices.

Responsibilities

• Lead technical architecture and solution design for large-scale transaction data platforms.
• Provide technical guidance on GCP architecture, data modelling, schema design and engineering best practices.
• Design and implement real-time and batch data ingestion and processing pipelines.
• Develop pipelines including Kafka to Spanner/BigQuery via Dataflow and GCS to Spanner/BigQuery.
• Independently develop, unit test and hand over production-ready data solutions.
• Perform pair programming and provide technical guidance to data engineering teams.
• Lead sprint planning, backlog management, scope definition, risks, issues and delivery timelines.
• Work as a technical workstream or overall project technical lead depending on project scale.
• Design solutions for migrating existing data platforms to GCP cloud-native services and for greenfield GCP implementations.
• Optimise BigQuery queries and data workloads for performance and scalability.
• Develop CI/CD pipelines for data workloads using Cloud Build, Artifact Registry and Terraform.
• Design orchestration workflows using Cloud Composer and Cloud Workflows.
• Support data governance and cataloguing using Dataplex and Data Catalog.
• Apply Generative AI technologies and integrations where appropriate within enterprise data solutions.

Skills

GCP data analytics

BigQuery

Dataflow

Kafka

Pub/Sub

Python

Java

Terraform

Cloud Build

Cloud Composer

Cloud Workflows

Dataplex

Data Catalog

CI/CD

Datalake/data platforms

Tools

Terraform

Cloud Build

Cloud Composer

Cloud Workflows

Dataplex

Data Catalog

Kafka

Pub/Sub

Python

Java

Job description

Delivery Centric Pty Ltd seeks a Senior GCP Data Architect and Data Engineering Lead to design and deliver large-scale real-time and batch data platforms on Google Cloud. You will combine solution architecture, hands-on data engineering, leadership and customer collaboration to drive complex data initiatives.

You will lead architecture, data modelling and pipelines, guiding teams on GCP-native services, Dataflow, BigQuery, Spanner and AI-enabled data solutions.