Senior Data Engineer

Comparethemarket; Brisbane; On-site; AUD 130,000 - 180,000; Full time

Benefits offered by this job

Vibrant and social community

Flexible work – one day WFH

‘ME’ leave and ‘Volunteer Day’ leave

Option to purchase additional leave

Return to Work bonus for parents

Quarterly Award and Recognition

Job summary

Compare the Market Pty is seeking a Senior Data Engineer to advance our cloud data platform (Databricks/AWS). You will lead hands‑on development and provide technical guidance, shaping architecture and data capabilities to support analytics and ML initiatives.

You will collaborate with data scientists, BI teams and architects to deliver scalable data solutions while upholding governance and security standards.

Qualifications

• 5+ years of experience building data pipelines in production environments.
• Strong Python and SQL skills for data engineering.
• Experience with cloud platforms (AWS) and modern data stacks.
• Knowledge of data modelling, governance, security and quality practices.
• Experience with CI/CD and Infrastructure as Code.

Responsibilities

• Build, maintain and optimise scalable data pipelines for analytics and ML.
• Design data solutions and contribute to platform architecture.
• Develop real‑time and batch data processing frameworks.
• Champion best practices across data modelling, quality and governance.
• Support and deliver against the data engineering roadmap.
• Collaborate with stakeholders to ensure data is accessible and reliable.

Skills

Data pipelines

Python

SQL

AWS

CI/CD

Infrastructure as Code

Mentoring

Tools

Databricks

Spark

PySpark

AWS

Job description

Who We Are

iSelect and Compare the Market are trusted, well-known brands dedicated to helping millions of Australians compare and buy personal finance and household products such as insurance, energy, and loans. Together, we provide a seamless, customer-focused experience that makes it easier for Australians to find the right products to suit their needs.

What You’ll Be Doing

As a Senior Data Engineer, you’ll play a key role in advancing our modern cloud data platform (Databricks/AWS). You’ll combine hands‑on engineering expertise with technical leadership, helping shape architecture, uplift data capabilities, and support advanced analytics and machine learning use cases.

You’ll work across a collaborative data ecosystem, partnering with data scientists, BI teams, and architects to deliver scalable, high‑quality data solutions.

• Build, maintain and optimise scalable data pipelines for analytics and ML
• Design data solutions and contribute to platform architecture
• Develop real‑time and batch data processing frameworks
• Champion best practices across data modelling, quality and governance
• Support and deliver against the data engineering roadmap
• Collaborate with stakeholders to ensure data is accessible and reliable

What’s In It For You

At Compare the Market and iSelect, we’re all about helping people make smart choices – and joining us might just be yours. We offer an environment where your ideas are heard, your contributions are valued, and your growth genuinely matters. Collaboration is part of our DNA, and you’ll be joining a supportive, forward‑thinking team where you can do meaningful work (and be recognised for it!)

We’re guided by three shared values that shape how we work, how we treat each other, and how we show up for our customers – Drive, Noble Intent and All Together.

Plus, we offer a range of great perks to help you thrive both in and out of work:

• Vibrant and social community with events and celebrations
• Flexible work – one day WFH or a 9‑day fortnight
• ‘ME’ leave and ‘Volunteer Day’ leave
• Option to purchase additional leave
• Return to Work bonus for parents
• Quarterly Award and Recognition program

What Sets You Apart

We’d love to hear from professionals who have:

• 5+ years’ experience building data pipelines in production environments
• Strong Python and SQL skills for data engineering
• Experience with cloud platforms (AWS) and modern data stacks
• Knowledge of data modelling, governance, security and quality practices
• Experience with CI/CD and Infrastructure as Code
• Ability to provide technical guidance and mentor others

Experience with the below would be advantageous:

• Databricks, Spark or PySpark experience
• Experience with real‑time/streaming data or ML feature stores
• Metadata‑driven or highly automated data solutions

Sound like a good fit? Let’s talk.

At Compare the Market and iSelect we’re proud to be an Equal Opportunity Employer and we embrace diversity.

This role is available to candidates with unlimited Australian working rights only. Sponsorship is not available.

iSelect and Compare the Market, brands of iCompare Group, are committed to protecting your privacy. All applications and personal information are kept confidential and securely stored in line with Australian privacy laws, used only for recruitment purposes, and not shared with third parties outside iCompare Group without your consent.