Senior Data Engineer - Cloud Data Platform & ML Pipelines

Compare The Market Pty; City of Brisbane; Hybrid; AUD 130,000 - 180,000; Full time

Benefits offered by this job

Vibrant and social community

Flexible work – one day WFH

‘ME’ leave and ‘Volunteer Day’ leave

Option to purchase additional leave

Return to Work bonus for parents

Quarterly Award and Recognition

Job summary

Compare the Market Pty is seeking a Senior Data Engineer to advance our cloud data platform (Databricks/AWS). You will lead hands‑on development and provide technical guidance, shaping architecture and data capabilities to support analytics and ML initiatives.

You will collaborate with data scientists, BI teams and architects to deliver scalable data solutions while upholding governance and security standards.

Qualifications

• 5+ years of experience building data pipelines in production environments.
• Strong Python and SQL skills for data engineering.
• Experience with cloud platforms (AWS) and modern data stacks.
• Knowledge of data modelling, governance, security and quality practices.
• Experience with CI/CD and Infrastructure as Code.

Responsibilities

• Build, maintain and optimise scalable data pipelines for analytics and ML.
• Design data solutions and contribute to platform architecture.
• Develop real‑time and batch data processing frameworks.
• Champion best practices across data modelling, quality and governance.
• Support and deliver against the data engineering roadmap.
• Collaborate with stakeholders to ensure data is accessible and reliable.

Skills

Data pipelines

Python

SQL

AWS

CI/CD

Infrastructure as Code

Mentoring

Tools

Databricks

Spark

PySpark

AWS

Job description

Compare the Market Pty is seeking a Senior Data Engineer to advance our cloud data platform (Databricks/AWS). You will lead hands‑on development and provide technical guidance, shaping architecture and data capabilities to support analytics and ML initiatives.

You will collaborate with data scientists, BI teams and architects to deliver scalable data solutions while upholding governance and security standards.