Data Engineer: Cloud Lakehouse & Databricks Expert

Compare the Market Australia; Brisbane; Presencial; AUD 120.000 - 180.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible work arrangements

One day work from home per week

9-day fortnight

ME leave

Volunteer Day leave

Free coffee on Mondays

Additional leave options

Return To Work bonus

Quarterly Award & Recognition program

Annual celebrations & events

Descripción de la vacante

Compare the Market Australia is seeking a Data Engineer to design, build and maintain scalable cloud data solutions using Databricks and related platforms, delivering robust data pipelines, lakehouse architectures and curated data assets to support reporting, analytics and AI/ML initiatives.

You will work with stakeholders, analysts, data scientists and engineers to transform raw data into trusted, accessible information while ensuring the platform remains governed, optimised and reliable.

Formación

• Degree in Computer Science, Analytics, IT or a related field, or equivalent practical experience.
• 3+ years in Data Engineering, or 5+ years across engineering, data or BI.
• 3+ years with cloud platforms such as AWS, Azure or GCP and Lakehouse technologies including Databricks, Snowflake, BigQuery, Redshift or Synapse.
• Building batch and near real-time pipelines using Spark, Python, SQL or similar languages.
• ETL/ELT, DataOps and orchestration tools such as DBT, DLT, Airflow, Dagster or Kestra.
• Cloud storage and Lakehouse architectures, including S3, ADLS or HDFS.
• Data modelling and governance, including Data Vault, metadata, lineage, auditability and access management.
• CI/CD, IaC, APIs, Azure DevOps, GitHub and automation frameworks.
• Enterprise Data Warehouse or Hadoop experience is advantageous.
• Exposure to GenAI, including RAG, AI Agents, Semantic/Vector Search, Knowledge Graphs, Embeddings, Fine-Tuning, Model Adaptation or AI Guardrails, is highly regarded.

Responsabilidades

• Design, build and maintain scalable data solutions aligned with enterprise architecture and governance standards.
• Build and optimise end-to-end data pipelines, integrations and workflows.
• Implement, test and deploy solutions using IaC, engineering best practices and release processes.
• Ensure data quality and reliability through monitoring, logging, issue remediation and proactive risk management.
• Partner with Solution Architects, Data Engineers, BI Developers and business stakeholders to optimise platforms and support analytics.
• Work effectively in an Agile environment, bringing strong communication, business knowledge and an end-to-end understanding of the Data & Insights lifecycle.
• Apply data engineering best practices and advance DataOps and automation to continually improve platform performance and scalability.

Conocimientos

Data engineering

Cloud data platforms

DataOps

CI/CD & IaC

Stakeholder collaboration

GenAI awareness

Educación

Degree in Computer Science, Analytics, IT or related field

Herramientas

Databricks

Snowflake

BigQuery

Redshift

Synapse

Spark

Python

SQL

DBT

DLT

Airflow

Dagster

Kestra

S3

ADLS

HDFS

Azure DevOps

GitHub

Descripción del empleo

Compare the Market Australia is seeking a Data Engineer to design, build and maintain scalable cloud data solutions using Databricks and related platforms, delivering robust data pipelines, lakehouse architectures and curated data assets to support reporting, analytics and AI/ML initiatives.

You will work with stakeholders, analysts, data scientists and engineers to transform raw data into trusted, accessible information while ensuring the platform remains governed, optimised and reliable.