Cloud Data Engineer: Spark, Azure & Lakehouse

Softtest Pays Pty; Sydney; On-site; AUD 130,000 - 170,000; Full time

Job summary

Softtest Pays Pty is seeking a Data Engineer in Sydney to design and implement cloud-based data pipelines. You will work on ETL/ELT processes for large-scale data movement and visualisations, leveraging Spark and cloud data services.

You will optimise data for query performance in distributed engines and collaborate with Engineering, Storage and Analytics teams to deploy data solutions in cloud infrastructure.

Qualifications

• Experience building ETL/ELT pipelines in cloud environments.
• Data modelling and query-optimised data layouts for analytics.
• Familiarity with Azure Data Factory and Databricks platforms.
• Knowledge of Data Lake and Lakehouse architectures.

Responsibilities

• Develop ETL/ELT pipelines for complex data movement, transformation and visualisation in the cloud (50%).
• Prepare data for fast queries in distributed/cloud engines (Spark, Azure SQL).
• Collaborate with Engineering, Storage and Analytics teams in cloud infrastructure.
• Engage with Azure Data Factory and Databricks for data integration.

Skills

Spark

Storage

Analytics

Infrastructure

Databases

Security

SQL

Engineering

Tools

Azure Data Factory

Databricks

Data Lake

Lakehouses

Job description

Softtest Pays Pty is seeking a Data Engineer in Sydney to design and implement cloud-based data pipelines. You will work on ETL/ELT processes for large-scale data movement and visualisations, leveraging Spark and cloud data services.

You will optimise data for query performance in distributed engines and collaborate with Engineering, Storage and Analytics teams to deploy data solutions in cloud infrastructure.