Head of Data Engineering

at; City of Brisbane; On-site; AUD 150,000 - 180,000; Full time

Job summary

A leading financial services company in Brisbane seeks a Head of Data Engineering to lead the development and operational excellence of core data platforms. You will drive best engineering practices, and oversee data optimization and automation. The ideal candidate possesses extensive experience with cloud platforms and big data frameworks. This dynamic role offers an opportunity to make a significant impact in an innovative team culture.

Benefits

Flexible work arrangements

Employee discounts on insurance

Additional leave days

Qualifications

• Demonstrated leadership with high-performing engineering teams.
• Ability to influence stakeholders and align data solutions.
• Expertise in big data frameworks and strong Python skills.

Responsibilities

• Lead development of data platforms and drive best practices.
• Define and deliver data engineering roadmap ensuring data quality.
• Implement DataOps for automation and testing.

Skills

Leadership

Team building

Cloud-based platforms (AWS, Azure, GCP)

Big data frameworks (Apache Spark, Hadoop)

Python

DataOps practices

Tools

Databricks

ETL/ELT tools (Terraform, dbt)

Orchestration tools (Airflow, Dagster)

Job description

iSelect and Compare the Market are trusted, well‑known brands dedicated to helping millions of Australians compare and buy personal finance and household products such as insurance, energy, and loans. Together, we provide a seamless, customer‑focused experience that makes it easier for Australians to find the right products to suit their needs.

What You’ll be doing

As Head of Data Engineering, you will lead the development, scalability, and operational excellence of our core data platforms. You’ll drive best‑in‑class engineering practices, ensure data integrity and availability, and oversee automation, monitoring, and optimisation of data workflows. You will also shape modern data architectures, implement cloud‑based big data solutions, and champion DataOps methodologies to deliver efficient, reliable, and innovative data capabilities.

• Define and deliver the data engineering roadmap, ensuring governance, security, and data quality.
• Architect, build and optimise scalable data solutions utilising Databricks as our Enterprise Date Platform.
• Implement DataOps practices for automation, CI/CD, and testing.
• Build and lead high‑performing teams while fostering collaboration and innovation.
• Stay ahead of emerging trends and adopt tools to enhance automation and cost efficiency.

What’s in it for you

At Compare the Market and iSelect we offer a dynamic and collaborative work environment where your talents will be recognised and rewarded. You’ll have the opportunity to make a meaningful impact and contribute to our continued success, we also provide:

• Vibrant and social community who have annual celebrations, family fun days and regular events.
• Enjoy flexible work arrangements, including the option of one day working from home each week or a 9‑day fortnight.
• Enjoy additional “ME” leave days.
• Employee discounts on Car, Home, Travel insurance.
• Income protection insurance provided to support you in the event of non‑work‑related illness or injury.

What Sets You Apart

• Demonstrated leadership and team‑building skills, with experience managing high‑performing engineering teams.
• Ability to develop strategy, influence senior stakeholders, and align data solutions with business objectives.
• Extensive experience with cloud‑based data platforms (AWS, Azure, GCP) and data lake/lakehouse architectures.
• Expertise in big data frameworks (Apache Spark, Hadoop, Snowflake, Redshift, BigQuery, Delta Lake) and strong skills in Python, PySpark, SQL.
• Proven ability to design and optimise ETL/ELT pipelines for structured and unstructured data.
• Hands‑on experience with DataOps practices, including CI/CD (Terraform, dbt, GitHub Actions, Azure DevOps) and orchestration tools (Airflow, Dagster, Prefect).
• Knowledge of real‑time streaming technologies (Kafka, Kinesis, Delta Live Tables).
• Strong understanding of data governance, security, and compliance (Unity Catalog, GDPR, CCPA, HIPAA), including RBAC, encryption, and data masking.

Sound like a good fit? Let’s talk.

At Compare the Market and iSelect we’re proud to be an Equal Opportunity Employer and we embrace diversity.

This role is available to candidates with unlimited Australian working rights only. Sponsorship is not available.

Compare the Market and iSelect are committed to protecting your privacy. All applications and personal information are kept confidential and securely stored in line with Australian privacy laws, used only for recruitment purposes, and not shared with third parties outside the Auto and General Holdings Group without your consent.