Senior Data Engineer- GCP & Cloud

CareCone Group; City of Melbourne; Presencial; AUD 140 000 - 220 000; Tempo integral

Resumo da oferta

CareCone Group seeks a senior data platform architect to lead the design and hands-on implementation of cloud-native data solutions on Google Cloud Platform. You will shape architecture and detailed designs, delivering scalable data pipelines for batch and real-time ingestion.

You will work with Confluent Kafka, GCS, Spanner and BigQuery, and drive CI/CD with Cloud Build and Terraform while enabling data governance with Dataplex and Data Catalog, including Generative AI integrations for

Qualificações

• Proficiency in Python and Java.
• Experience with GCP data and analytics technologies.
• Hands-on software development and cloud data engineering skills.

Responsabilidades

• Design large-scale transaction data platform architecture for migrating open-source or public-cloud data platforms to GCP native services.
• Define solution architecture and detailed design, and perform hands-on implementation for data pipelines (batch and event-driven).
• Build data pipelines ingesting from on-prem files, on-prem databases, and APIs.
• Implement real-time ingestion and event processing from Confluent Kafka and GCS to Spanner and BigQuery.
• Create CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform.
• Set up orchestration and Cloud Composer DAG development for data pipelines.
• Apply data governance using Dataplex and Data Catalog.
• Explore Generative AI integrations in enterprise environments.

Conhecimentos

Python

Java

GCP data analytics

Ferramentas

Cloud Build

Artifact Registry

Terraform

Cloud Composer

Dataplex

Data Catalog

Confluent Kafka

BigQuery

Spanner

GCS

Descrição da oferta de emprego

Job Type- Contract

Location- Sydney/Melbourne

JD-

Must have

• Must have: programming knowledge and willingness to be hands-on - Python, Java

In-depth knowledge and experience of GCP data and analytics technologies

• Designing Large scale Transaction Data Platform architecture for (a) migrating open source or public cloud data platforms to GCP cloud native services (b) designing greenfield data platforms on GCP
• Define solution architecture and detailed design, and perform hands-on implementation for –

data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.

real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery

• Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
• CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
• Orchestration setup and Cloud Composer DAG development for data pipeline workflows
• Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog)
• Experience applying Generative AI technologies and integrations in enterprise environments.