Data Engineers

Softtest Pays Pty; Darwin; On-site; AUD 110,000 - 160,000; Full time

Job summary

Softtest Pays Pty is seeking a data engineer to design and implement ETL/ELT processes for large data movements in cloud environments. You will optimize data for query performance in engines like Spark and Azure SQL, and collaborate with engineering, storage and analytics teams.

Experience with Azure Data Factory and Databricks, as well as data lake/lakehouse concepts, is highly valued. Security clearances and Australian eligibility are required.

Qualifications

• Experience developing ETL/ELT processes for large data movement in cloud environments.
• Experience preparing data for query performance in cloud compute engines (Spark, Azure SQL).
• Experience with engineering, storage and analytics services in cloud infrastructure.
• Experience with Azure Data Factory and Databricks.
• Experience/Knowledge of Data Lake and Lakehouses.

Skills

ETL/ELT development

Cloud data performance

Tools

Azure Data Factory

Databricks

Spark

Graph Databases

Azure SQL

Job description

Australian Citizens residing in Australia only respond . must hold a valid Baseline security clearance or be willing and eligible to obtain one.

1. Demonstrated experience developing ETL/ELT processes for complex and/or large data movement, transformation and/or visualisations, particularly in a cloud environment. 50%

2. Experience preparing data optimised for query performance in cloud computed engines. E.g. Distributed computing engines (Spark) Graph Databases Azure SQL. 25%

3. Experience working with Engineering, Storage and Analytics services in cloud infrastructure. 25%

1. Experience working with Azure Data Factory and Databricks.

2. Experience/Knowledge of working with Data Lake and Lakehouses

Required Skills:

Spark Storage Analytics Infrastructure Databases Security SQL Engineering