Data Engineer

TaskVerse; Australia; On-site; AUD 120,000 - 180,000; Full time

Job summary

Softtest Pays Pty is seeking an experienced Data Engineer to design, develop and optimize ETL/ELT processes for large-scale data workloads in a cloud environment. You will work across ingestion, transformation and query-ready outputs to ensure data is structured for cloud analytics.

Applicants should have practical cloud data platform experience and familiarity with Spark, Azure Data Factory, Databricks and data lakes or lakehouses.

Qualifications

• Experience developing ETL/ELT for complex or large-scale data workloads.
• Strong ability to prepare data for query performance in cloud compute engines.
• Familiarity with SQL and data platform concepts; Azure Data Factory and Databricks knowledge is preferred.

Responsibilities

• Design, develop, test and maintain ETL/ELT workflows across cloud platforms.
• Prepare and organize data to improve query performance in cloud environments.
• Collaborate with engineering, storage and analytics teams to deliver integrated data solutions.

Skills

ETL/ELT

Cloud data platforms

SQL

Spark

Education

Bachelor’s degree in CS or related

Tools

Azure Data Factory

Databricks

Azure SQL

Spark

Job description

About The Company

Softtest Pays Pty works with organisations to connect skilled technology professionals with complex project opportunities. This role supports data engineering initiatives in a cloud environment, contributing to the movement, transformation, storage and preparation of data for analysis and visualisation. The successful candidate will work with engineering, storage and analytics services to help deliver reliable, usable data assets for project stakeholders.

About The Company

Softtest Pays Pty works with organisations to connect skilled technology professionals with complex project opportunities. This role supports data engineering initiatives in a cloud environment, contributing to the movement, transformation, storage and preparation of data for analysis and visualisation. The successful candidate will work with engineering, storage and analytics services to help deliver reliable, usable data assets for project stakeholders.

About The Role

Softtest Pays Pty is seeking an experienced Data Engineer to develop and improve ETL and ELT processes for complex and large-scale data workloads. You will work across the data lifecycle, from ingestion and transformation through to query-ready outputs, helping ensure data is structured and optimised for cloud-based processing and analytics.
The role requires practical experience with cloud data platforms and distributed computing, including the ability to prepare data for efficient querying in cloud compute engines. You will collaborate with engineering, storage and analytics teams to understand requirements, develop appropriate solutions and support dependable data services. Experience with Azure Data Factory, Databricks, data lakes or lakehouses is desirable.
This opportunity is available to Australian citizens residing in Australia. Applicants must hold a current Baseline security clearance or be willing and eligible to obtain one. The position may involve work with sensitive information, so sound judgement, attention to detail and a responsible approach to security are important.

Qualifications

Candidates should demonstrate experience developing ETL or ELT processes for complex or large-scale data movement, transformation or visualisation, particularly in cloud environments. Strong capability in preparing data for query performance in cloud compute engines is essential. Relevant experience may include distributed computing technologies such as Spark, graph databases, Azure SQL or comparable platforms.
You should also have experience working with cloud infrastructure services across data engineering, storage and analytics. The ability to translate technical requirements into maintainable data solutions, investigate issues and work effectively with multidisciplinary teams will be valuable. Familiarity with SQL and data platform concepts is expected. Experience with Azure Data Factory and Databricks, as well as knowledge of data lakes and lakehouses, will be viewed favourably.

Responsibilities

Design, develop, test and maintain ETL and ELT workflows that move and transform data across cloud platforms. Build processes that can handle complex datasets and large volumes while supporting reliable downstream reporting, analytics or visualisation.
Prepare and organise data to improve query performance in cloud-based compute environments. Apply suitable approaches to distributed processing, storage and data modelling, and contribute to solutions involving technologies such as Spark, Azure SQL or graph databases where appropriate.
Work with engineering, storage and analytics services to deliver integrated data solutions. Collaborate with technical colleagues and stakeholders to clarify needs, assess data quality, troubleshoot pipeline or performance issues, and document processes. Follow applicable security practices when accessing, processing or managing data, including requirements associated with a Baseline-cleared environment.

Benefits

This role offers the opportunity to contribute to substantial cloud data initiatives and solve practical engineering challenges involving data scale, transformation and query performance. You will work across interconnected engineering, storage and analytics capabilities, with scope to apply and further develop skills in cloud data services, distributed computing and modern data platforms. The position may also provide exposure to Azure Data Factory, Databricks, data lakes and lakehouse approaches, depending on project requirements.

Equal Opportunity

Softtest Pays Pty is committed to fair and respectful recruitment. Applications are considered on relevant skills, experience and eligibility for the role. Applicants must be Australian citizens residing in Australia and must hold, or be willing and eligible to obtain, a Baseline security clearance. All candidates are encouraged to apply if they meet these requirements and can demonstrate the capabilities described above.