Senior Big Data Engineer – Real-Time Analytics & Spark

Pyramid Global Technologies; New South Wales; On-site; AUD 120,000 - 180,000; Full time

Job summary

Pyramid Global Technologies in New South Wales, Australia seeks an experienced Data Engineer to join our analytics team and drive data engineering and warehousing initiatives. You will work across ingestion, processing, and storage layers using Spark, Kafka, Hive, and related technologies to build scalable analytics platforms.

Strong SQL, ETL, and data profiling skills are essential to deliver reliable data pipelines.

Qualifications

• 5+ years experience working in Data Engineering and Warehousing.
• 3-5 years’ experience integrating data into analytical platforms.
• Experience in ingestion technologies (e.g. Sqoop, flume), processing technologies (Spark/Scala), and storage (e.g. HDFS, HBase, Hive).
• Experience in data profiling, source-target mappings, ETL development, SQL optimization, testing, and implementation.
• Expertise in streaming frameworks (Kafka/Spark Streaming/Storm) essential.
• Experience building Spark Sql and Spark Data Frame API-based applications.
• Experience managing structured and unstructured data types.
• Experience in requirements engineering, solution architecture, design, and development/deployment.
• Experience in creating big data or analytics IT solution.
• Track record of implementing databases and data access middleware and high-volume batch and near real-time processing.

Skills

Data Engineering

Warehousing

ETL Development

SQL Optimization

Spark

Kafka

Spark SQL

Spark DataFrame API

Streaming Frameworks

HDFS

Hive

Data Profiling

Source-Target Mappings

Solution Architecture

Big Data Platform Experience

Job description

Pyramid Global Technologies in New South Wales, Australia seeks an experienced Data Engineer to join our analytics team and drive data engineering and warehousing initiatives. You will work across ingestion, processing, and storage layers using Spark, Kafka, Hive, and related technologies to build scalable analytics platforms.

Strong SQL, ETL, and data profiling skills are essential to deliver reliable data pipelines.