Senior Data Engineer

iSelect; City of Brisbane; On-site; AUD 150,000 - 210,000; Full time

Benefits offered by this job

Flexible work arrangements

ME leave and Volunteer Day leave

Free coffee on Mondays

9-day fortnight

Annual celebrations and events

Job summary

iSelect is seeking a Senior Data Engineer to design and evolve a cloud-based data platform that supports BI, analytics, ML, and Generative AI initiatives. You will lead data pipelines, governance, and platform optimisation while collaborating with business and tech stakeholders.

You will champion DataOps, automation, CI/CD, and secure data access, delivering scalable data products and AI-ready datasets for model development and deployment across AWS/Azure/GCP ecosystems.

Qualifications

• Bachelor’s degree in Computer Science, Analytics, IT or a relevant discipline.
• 7+ years in Data Engineering/Analytics/DBA/BI with enterprise-scale data solutions.
• 5+ years designing and supporting cloud data platforms and data products across AWS/Azure/GCP.
• Strong data architectures: Data Warehouse, Lakehouse, Data Mesh, Data Vault, metadata, lineage & data quality.

Responsibilities

• Design, develop, and support scalable cloud-based data warehouse, lakehouse, and AI-enabled data platform solutions for analytics and future ML/AI initiatives.
• Build and maintain batch, streaming, and near real-time data pipelines and data products with high quality, governed data.
• Develop data workflows using AI agents, orchestration, and modern AI platform capabilities to improve operations.
• Support AI initiatives by delivering AI-ready datasets, vector search, and secure scalable data solutions for model development and deployment.
• Implement DataOps, SecOps, CI/CD, IaC, observability, and data security controls across platforms.
• Ensure governance, security, privacy, and quality standards while improving platform capabilities and cost efficiency.

Skills

Data engineering

Python

SQL

Apache Spark

PySpark

DataOps

MLOps

ETL/ELT

CI/CD

Cloud data warehouses

Education

Bachelor's degree in CS/Analytics

Tools

Databricks

Snowflake

Redshift

Kinesis

Kafka

Fabric

Synapse

BigQuery

Job description

Who We Are

iSelect and Compare the Market are trusted, well-known brands dedicated to helping millions of Australians compare and buy personal finance and household products such as insurance, energy, and loans. Together, we provide a seamless, customer-focused experience that makes it easier for Australians to find the right products to suit their needs.

What You’ll Be Doing

Reporting to the Data Engineering Manager, the Senior Data Engineer designs, builds, and evolves the organisation’s cloud data platform, delivering trusted, scalable, and secure data solutions that support business intelligence, advanced analytics, machine learning, and AI initiatives.

The role develops and optimises cloud-based data pipelines, products, and platform services, ensuring data is accessible, governed, and fit for reporting, analytics, predictive modelling, and Generative AI use cases.

Working closely with stakeholders across the business and technology teams, the Senior Data Engineer helps transform data into strategic assets that drive decision-making and innovation, while supporting AI-enabled solutions through strong data foundations and governance.

As a senior technical leader, the role provides expertise across data architecture, engineering standards, platform optimisation, automation, and DataOps/MLOps practices, contributing to the ongoing maturity of the organisation’s data and AI capabilities.

• Design, develop, and support scalable cloud-based data warehouse, lakehouse, and AI-enabled data platform solutions that support enterprise analytics, reporting and future ML/AI initiatives


• Build, optimise, and maintain batch, streaming, and near real-time data pipelines and data products to deliver trusted, high-quality, and governed data for business, analytics, and up-coming AI workloads


• Design, develop and deploy data workflows as an individual contributor with support of intelligent automation solutions which leverages AI agents, agentic workflows, LLMs, orchestration frameworks, and modern AI platform capabilities to improve operational efficiency and business outcomes


• Support advanced analytics and AI initiatives through the delivery of AI-ready datasets, vector search capabilities, RAG architectures, and secure, scalable data solutions that enable model development and operationalisation


• Implement and champion modern engineering practices including DataOps, SecOps, CI/CD, IaC automated testing, observability, platform automation, and responsible data access controls


• Ensure compliance with data governance, security, privacy, and quality standards, while collaborating with business and technology stakeholders to continuously improve platform capability, innovation, scalability, performance, and cost efficiency

What’s In It For You

At Compare the Market and iSelect, we’re all about helping people make smart choices – and joining us might just be yours.We offer an environment where your ideas are heard, your contributions are valued, and your growth genuinely matters. Collaboration is part of our DNA and you’ll be joining a supportive, forward-thinking team where you can do meaningful work (and be recognised for it!)

We’re guided by three shared values that shape how we work, how we treat each other, and how we show up for our customers - Drive, Noble Intent and All Together.

Plus, we offer a range of great perks to help you thrive both in and out of work:

• Vibrant and social community who have annual celebrations, family fun days and regular events


• Enjoy flexible work arrangements, including the option of one day working from home each week or a 9-day fortnight


• Enjoy additional leave days - ‘ME’ leave and ‘Volunteer Day’ leave


• Free coffee on Mondays


• Option to purchase additional leave


• Return To Work bonus for returning parents


• Quarterly Award and Recognition program

What Sets You Apart

Qualifications

• Minimum a bachelor’s degree qualified in Computer Science, Analytics, IT or a relevant discipline

Experience & Skills

• 7+ years' experience in Data Engineering, Analytics, Database Administration, Software Engineering, or Business Intelligence environments, with a proven track record delivering enterprise-scale data solutions


• 5+ years' experience designing, building, and supporting cloud-based data platforms, data warehouses, lakehouses, and data products across AWS, Azure, or GCP environments


• Strong expertise in modern data architectures and platforms, including Data Warehouses, Data Lakes, Lakehouses, Data Mesh, Digital Twin, Medallion Architectures, and platforms such as Databricks, Snowflake, Fabric, Synapse, Redshift, or BigQuery


• Extensive experience developing batch, streaming, and near real-time data pipelines using Apache Spark, PySpark, SQL, Kafka, Kinesis, Event Hub, and cloud-native storage and distributed data platforms


• Advanced proficiency in Python, SQL, and PySpark, with experience in additional programming languages such as Scala, Java, R, TypeScript, or Julia


• Strong experience delivering DataOps and software engineering practices, including ETL/ELT frameworks, orchestration platforms, CI/CD, IaC, DevOps tooling, automated testing, observability, monitoring, and platform automation


• Proven expertise in data modelling, governance, and data management, including dimensional modelling, Data Vault, metadata management, lineage, data quality, cataloguing, master data, access controls, and audit requirements


• Experience developing reusable frameworks, APIs, event-driven architectures, integration services, and metadata-driven automation solutions that improve scalability, maintainability, and operational efficiency


• Experience supporting current and future advanced analytics, ML and Gen AI initiatives, including AI-ready datasets, MLOps, AI platform integration, vector search, RAG, semantic search, embeddings, knowledge retrieval, and intelligent automation solutions


• Demonstrated technical leadership and stakeholder engagement, including mentoring engineers, contributing to solution architecture, establishing engineering standards, integrating enterprise and SaaS platforms, and applying security, privacy, compliance, and responsible data governance principles

At Compare the Market and iSelect we’re proud to be an Equal Opportunity Employer and we embrace diversity.

This role is available to candidates with unlimited Australian working rights only. Sponsorship is not available.

iSelect and Compare the Market, brands of iCompare Holdings, are committed to protecting your privacy. All applications and personal information are kept confidential and securely stored in line with Australian privacy laws, used only for recruitment purposes, and not shared with third parties outside iCompare Holdings without your consent.

At Compare the Market and iSelect we’re proud to be an Equal Opportunity Employer and we embrace diversity.

This role is available to candidates with unlimited Australian working rights only. Sponsorship is not available.

iSelect and Compare the Market, brands of iCompare Group, are committed to protecting your privacy. All applications and personal information are kept confidential and securely stored in line with Australian privacy laws, used only for recruitment purposes, and not shared with third parties outside iCompare Group without your consent.