Senior Data Engineer - GCP Cloud-Native Pipelines (Hybrid)

PBT Group; Johannesburg; Hybrid; ZAR 900,000 - 1,500,000; Full time

Job summary

PBT Group is seeking an experienced Senior Data Engineer with strong Google Cloud Platform (GCP) expertise to join a project focused on designing, developing and optimising scalable, cloud-native data platforms and near-real-time data pipelines.

The successful candidate will have extensive experience building highly parallelised, distributed data-processing solutions, with a strong understanding of batch and streaming architectures, event-driven processing and cloud-based data engineering.

Qualifications

• 8+ years of experience in Data Engineering.
• Strong GCP experience with BigQuery, Dataflow, Cloud Composer, and GKE.
• Proficient in Python and SQL.
• Experience designing distributed, scalable data pipelines (batch and streaming).
• Experience with dbt and cloud-native architectures.

Responsibilities

• Design, develop and maintain scalable cloud-native data pipelines on GCP.
• Build and optimise batch and near-real-time data pipelines.
• Implement distributed data-processing solutions with parallelism.
• Develop workloads using Cloud Run and GKE.
• Design streaming architectures with Pub/Sub and Dataflow.
• Work with large-scale datasets to optimise performance and cost.
• Maintain data workflows and orchestration solutions.
• Collaborate with technical teams and stakeholders to deliver robust data solutions.

Skills

Data engineering

GCP expertise

Python

SQL

Distributed processing

Batch & streaming

Education

Relevant degree in CS/IT/Engineering

Google Cloud Professional Data Engineer certification (advantageous)

Tools

BigQuery

Dataflow

Cloud Run

GKE

Pub/Sub

Cloud Composer

Dataproc

Job description

PBT Group is seeking an experienced Senior Data Engineer with strong Google Cloud Platform (GCP) expertise to join a project focused on designing, developing and optimising scalable, cloud-native data platforms and near-real-time data pipelines.

The successful candidate will have extensive experience building highly parallelised, distributed data-processing solutions, with a strong understanding of batch and streaming architectures, event-driven processing and cloud-based data engineering.