Senior Data Engineer: Azure, PySpark & Big Data

Pyramid Global Technologies; Western Australia; On-site; AUD 140,000 - 180,000; Full time

Job summary

Pyramid Global Technologies is seeking a Senior Data Engineer for a Perth/Australia based role focused on Azure cloud data pipelines and big data processing. The candidate will design and implement data integration flows, leveraging Spark, Hive, HBase, Kafka and related technologies to load and transform large datasets.

Desirable experience includes ETL on cloud or on-prem platforms, strong SQL skills, and familiarity with Databricks, Hadoop/Spark using Java/Scala.

Qualifications

• Strong Data Engineering skills using Azure and Pyspark (or Databricks, Hadoop/Spark using Java/Scala)
• Experience in Azure Data Factory and other Azure services
• Experience in loading and transforming the data using spark or any big data technologies( Hive, Kafka, Hbase ,Spark or Storm)
• Very good SQL knowledge
• ETL Process experience in any Cloud or any on-premise big data platform.

Skills

Data Engineering

SQL

Pyspark

Databricks

Hadoop/Spark

Java/Scala

Azure

Tools

Azure Data Factory

Spark

Hive

HBase

Kafka

Storm

Databricks

Java

Scala

Job description

Pyramid Global Technologies is seeking a Senior Data Engineer for a Perth/Australia based role focused on Azure cloud data pipelines and big data processing. The candidate will design and implement data integration flows, leveraging Spark, Hive, HBase, Kafka and related technologies to load and transform large datasets.

Desirable experience includes ETL on cloud or on-prem platforms, strong SQL skills, and familiarity with Databricks, Hadoop/Spark using Java/Scala.