Data Engineers

Softtest Pays Pty; Sydney; On-site; AUD 130,000 - 170,000; Full time

Job summary

Softtest Pays Pty is seeking a Data Engineer in Sydney to design and implement cloud-based data pipelines. You will work on ETL/ELT processes for large-scale data movement and visualisations, leveraging Spark and cloud data services.

You will optimise data for query performance in distributed engines and collaborate with Engineering, Storage and Analytics teams to deploy data solutions in cloud infrastructure.

Qualifications

• Experience building ETL/ELT pipelines in cloud environments.
• Data modelling and query-optimised data layouts for analytics.
• Familiarity with Azure Data Factory and Databricks platforms.
• Knowledge of Data Lake and Lakehouse architectures.

Responsibilities

• Develop ETL/ELT pipelines for complex data movement, transformation and visualisation in the cloud (50%).
• Prepare data for fast queries in distributed/cloud engines (Spark, Azure SQL).
• Collaborate with Engineering, Storage and Analytics teams in cloud infrastructure.
• Engage with Azure Data Factory and Databricks for data integration.

Skills

Spark

Storage

Analytics

Infrastructure

Databases

Security

SQL

Engineering

Tools

Azure Data Factory

Databricks

Data Lake

Lakehouses

Job description

Australian Citizens residing in Australia only respond

must hold a valid Baseline security clearance or be willing and eligible to obtain one.

Key Responsibilities

• 1. Demonstrated experience developing ETL/ELT processes for complex and/or large data movement, transformation and/or visualisations, particularly in a cloud environment. 50%


• 2. Experience preparing data optimised for query performance in cloud computed engines. E.g. Distributed computing engines (Spark) Graph Databases Azure SQL. 25%


• 3. Experience working with Engineering, Storage and Analytics services in cloud infrastructure. 25%


• 1. Experience working with Azure Data Factory and Databricks.


• 2. Experience/Knowledge of working with Data Lake and Lakehouses

Required Skills:

Spark Storage Analytics Infrastructure Databases Security SQL Engineering

Data Engineers Sydney, New South Wales, Australia