Senior GCP Data Architect / Data Engineering Lead – Sydney/Melbourne

Delivery Centric Pty Ltd; Sydney; Presencial; AUD 180 000 - 250 000; Tempo integral

Resumo da oferta

Delivery Centric Pty Ltd seeks a Senior GCP Data Architect and Data Engineering Lead to design and deliver large-scale real-time and batch data platforms on Google Cloud. You will combine solution architecture, hands-on data engineering, leadership and customer collaboration to drive complex data initiatives.

You will lead architecture, data modelling and pipelines, guiding teams on GCP-native services, Dataflow, BigQuery, Spanner and AI-enabled data solutions.

Qualificações

• Strong hands-on experience with Google Cloud Platform data and analytics technologies.
• Strong experience with GCP Dataflow, BigQuery and Cloud Spanner.
• Experience with real-time/event-driven data ingestion using Confluent Kafka and Pub/Sub.
• Experience with Cloud Composer and Cloud Workflows.
• Strong experience with Terraform and Cloud Build for data engineering workloads.
• Hands-on programming experience with Python and Java.
• Experience designing and implementing batch and event-driven data pipelines from sources including databases, files, APIs and streaming platforms.
• Experience with BigQuery performance optimisation.
• Experience with CI/CD, infrastructure automation and production data workload deployment.
• Strong understanding of data architecture, solution design and engineering best practices.

Responsabilidades

• Lead technical architecture and solution design for large-scale transaction data platforms.
• Provide technical guidance on GCP architecture, data modelling, schema design and engineering best practices.
• Design and implement real-time and batch data ingestion and processing pipelines.
• Develop pipelines including Kafka to Spanner/BigQuery via Dataflow and GCS to Spanner/BigQuery.
• Independently develop, unit test and hand over production-ready data solutions.
• Perform pair programming and provide technical guidance to data engineering teams.
• Lead sprint planning, backlog management, scope definition, risks, issues and delivery timelines.
• Work as a technical workstream or overall project technical lead depending on project scale.
• Design solutions for migrating existing data platforms to GCP cloud-native services and for greenfield GCP implementations.
• Optimise BigQuery queries and data workloads for performance and scalability.
• Develop CI/CD pipelines for data workloads using Cloud Build, Artifact Registry and Terraform.
• Design orchestration workflows using Cloud Composer and Cloud Workflows.
• Support data governance and cataloguing using Dataplex and Data Catalog.
• Apply Generative AI technologies and integrations where appropriate within enterprise data solutions.

Conhecimentos

GCP data analytics

BigQuery

Dataflow

Kafka

Pub/Sub

Python

Java

Terraform

Cloud Build

Cloud Composer

Cloud Workflows

Dataplex

Data Catalog

CI/CD

Datalake/data platforms

Ferramentas

Terraform

Cloud Build

Cloud Composer

Cloud Workflows

Dataplex

Data Catalog

Kafka

Pub/Sub

Python

Java

Descrição da oferta de emprego

Senior GCP Data Architect / Data Engineering Lead – Sydney/Melbourne

Delivery Centric is seeking a Senior GCP Data Architect / Data Engineering Lead to design and deliver large-scalereal-time and batch transaction data platforms on Google Cloud. The role combines solution architecture, hands-on data engineering, technical leadership and customer collaboration.

Key Responsibilities

• Lead technical architecture and solution design for large-scale transaction data platforms.
• Provide technical guidance onGCP architecture, data modelling, schema design and engineering best practices.
• Design and implement real-time and batch data ingestion and processing pipelines.
• Develop pipelines includingKafka to Spanner/BigQuery via Dataflow and GCS to Spanner/BigQuery.
• Independently develop, unit test and hand over production-ready data solutions.
• Perform pair programming and provide technical guidance to data engineering teams.
• Lead sprint planning, backlog management, scope definition, risks, issues and delivery timelines.
• Work as a technical workstream or overall project technical lead depending on project scale.
• Design solutions for migrating existing data platforms toGCP cloud-native services and for greenfield GCP implementations.
• Optimise BigQuery queries and data workloads for performance and scalability.
• Develop CI/CD pipelines for data workloads usingCloud Build, Artifact Registry and Terraform.
• Design orchestration workflows usingCloud Composer and Cloud Workflows.
• Support data governance and cataloguing usingDataplex and Data Catalog.
• Apply Generative AI technologies and integrations where appropriate within enterprise data solutions.

Must-Have Skills

• Strong hands-on experience withGoogle Cloud Platform data and analytics technologies.
• Strong experience with GCP Dataflow, BigQuery and Cloud Spanner.
• Experience with real-time/event-driven data ingestion usingConfluent Kafka and Pub/Sub.
• Experience with Cloud Composer and Cloud Workflows.
• Strong experience with Terraform and Cloud Build for data engineering workloads.
• Hands-on programming experience withPython and Java.
• Experience designing and implementing batch and event-driven data pipelines from sources including databases, files, APIs and streaming platforms.
• Experience with BigQuery performance optimisation.
• Experience with CI/CD, infrastructure automation and production data workload deployment.
• Strong understanding of data architecture, solution design and engineering best practices.

Good-to-Have Skills

• DBT or Dataform.
• Experience with Cloud Run and enterprise GCP services.
• Experience with Dataplex and Data Catalog.
• Experience modernising or migrating legacy data platforms to GCP.
• Experience with legacy technologies such asHadoop, Spark, Cloudera/Hortonworks, HBase and Oozie.
• Experience with NoSQL databases such as HBase or MongoDB.
• Knowledge of Spanner, Bigtable, Cloud SQL and database migration approaches.
• Experience modernising legacy data-processing logic such asHive, Teradata and Python into BigQuery SQL.
• Experience applying Generative AI technologies within enterprise environments.
• GCP Data Engineering certification.

Key Skills

GCP | Dataflow | BigQuery | Cloud Spanner | Pub/Sub | Confluent Kafka | Cloud Composer | Cloud Workflows | Terraform | Cloud Build | Python | Java | Real-Time Data | Batch Data | Transaction Data Platforms | Data Architecture | Data Engineering| Technical Leadership