Data Engineer: Cloud Pipelines & Lakehouse, Flexible Work

Compare the Market Australia; Brisbane; Hybrid; AUD 120,000 - 170,000; Full time

Benefits offered by this job

Flexible work arrangements

ME leave and Volunteer Day leave

Annual celebrations and family events

Free coffee on Mondays

Job summary

Compare the Market Australia (iSelect and Compare the Market) is hiring a Data Engineer to design, build and maintain scalable cloud data solutions. You will deliver robust data pipelines, lakehouse architectures and automation frameworks to support analytics and AI/ML initiatives.

Join a collaborative team that embraces flexible work arrangements and values innovation, data governance, and cross-functional partnership to drive impactful insights.

Qualifications

• Degree in Computer Science, Analytics, IT or related field, or equivalent practical experience.
• 3+ years in Data Engineering or 5+ years across engineering, data or BI.
• 3+ years with cloud platforms and lakehouse technologies (Databricks, Snowflake, BigQuery, Redshift or Synapse).
• Experience building batch and near real-time pipelines using Spark, Python or SQL.
• Proficiency with ETL/ELT, DataOps and orchestration tools (DBT, Airflow, Dagster, Kestra).
• Knowledge of data modelling, governance, lineage, and access management.
• CI/CD, IaC, APIs and automation frameworks (Azure DevOps, GitHub).
• Exposure to GenAI and AI-related data workflows is a bonus.

Responsibilities

• Design and deliver scalable data solutions aligned with enterprise architecture.
• Build and optimise end-to-end data pipelines and integrations.
• Implement, test and deploy solutions using IaC and modern release processes.
• Ensure data quality and reliability through monitoring and logging.
• Collaborate with architects, BI developers and stakeholders to optimise platforms.
• Work in an Agile environment with strong communication and data lifecycle understanding.
• Apply DataOps and automation to improve performance and scalability.

Skills

3+ years in Data Engineering

5+ years across engineering, data orBI

Batch and near real-time pipelines

ETL/ELT and DataOps

CI/CD, IaC and automation

Data modelling, governance, lineage

Spark, Python, SQL

Cloud platforms (AWS/Azure/GCP)

Databricks, lakehouse architectures

GenAI exposure (optional)

Education

Degree in Computer Science, Analytics, IT or related field

Tools

Databricks

Snowflake

BigQuery

Redshift

Synapse

Airflow

DBT

DLT

Dagster

Kestra

Azure DevOps

GitHub

Job description

Compare the Market Australia (iSelect and Compare the Market) is hiring a Data Engineer to design, build and maintain scalable cloud data solutions. You will deliver robust data pipelines, lakehouse architectures and automation frameworks to support analytics and AI/ML initiatives.

Join a collaborative team that embraces flexible work arrangements and values innovation, data governance, and cross-functional partnership to drive impactful insights.