Lead Data Engineer
Compare the Market Australia
A complete application in a minute
Tailored resume and cover letter, ready to send.
Benefits offered by this job
Job summary
Compare the Market Australia is seeking a Lead Data Engineer to provide technical leadership across Data & Analytics, shaping the enterprise data stack and standards.
You will govern cloud-based data warehouses/lakehouses, drive modern engineering practices, and mentor teams to deliver high-quality data products and AI-enabled capabilities.
Qualifications
- Bachelor's degree in Computer Science, Analytics, IT or a relevant discipline.
- 10+ years' experience in Data Engineering, Analytics, Software Engineering, or BI environments.
- Experience designing cloud data platforms across AWS, Azure, or GCP.
- Strong data architecture and platform design skills (DWH, Data Lakes, Lakehouses).
- Advanced experience with modern data platforms like Databricks, Snowflake, and cloud services.
Responsibilities
- Provide technical leadership for Data & Analytics, aligning with enterprise architecture and security.
- Own and optimise cloud data platforms, data integration, governance, and AI readiness.
- Lead design and delivery of scalable data products and GenAI-enabled solutions.
- Drive DataOps, SecOps, CI/CD, IaC, automated testing, observability, and platform automation.
- Mentor engineers and ensure data quality, lineage, privacy, and regulatory compliance.
Skills
Education
Tools
Job description
iSelect and Compare the Market are trusted, well-known brands dedicated to helping millions of Australians compare and buy personal finance and household products such as insurance, energy, and loans. Together, we provide a seamless, customer-focused experience that makes it easier for Australians to find the right products to suit their needs.
iSelect and Compare the Market are trusted, well-known brands dedicated to helping millions of Australians compare and buy personal finance and household products such as insurance, energy, and loans. Together, we provide a seamless, customer-focused experience that makes it easier for Australians to find the right products to suit their needs.
Reporting to the Data Engineering Manager, the Lead Data Engineer provides technical leadership for CTM’s Data and Analytics platforms, engineering practices, and technology capabilities. The role is responsible for shaping the enterprise data technology stack, establishing engineering standards, and delivering scalable, secure, and high-performing data solutions that support reporting, analytics, AI, and business innovation.
As the organisation’s technical SME for data engineering, the Lead Data Engineer drives the design, governance, and optimisation of cloud-based data warehouse, lakehouse, and AI-enabled platforms. Working closely with engineers, analysts, architects, and business stakeholders, the role champions modern engineering practices, platform automation, DataOps, MLOps, and AI engineering capabilities to ensure the data ecosystem remains robust, future-ready, and aligned to strategic objectives.
The role oversees the technical health, security, performance, scalability, and continuous improvement of the data platform landscape while fostering a culture of engineering excellence, innovation, and knowledge sharing. It also provides technical mentorship and governance across the Data and Analytics team, guiding solution design, architecture standards, and development practices to ensure the consistent delivery of high-quality data products and services.
- Provide technical leadership and architectural oversight across the Data & Analytics function, ensuring solutions align with enterprise architecture, security, scalability, and business objectives.
- Own and optimise the analytics ecosystem, including cloud data platforms, lakehouse environments, data integration, governance, and AI readiness.
- Lead the design and delivery of scalable data products, advanced analytics, and GenAI-enabled solutions that support self-service insights and future AI capabilities.
- Drive engineering excellence through standards, frameworks, DataOps, SecOps, CI/CD, Infrastructure as Code, automated testing, observability, and automation.
- Ensure platform reliability, security, governance, and compliance through robust controls for data quality, lineage, metadata, access management, monitoring, and privacy.
- Mentor engineers and business users while partnering with stakeholders to drive innovation, platform optimisation, emerging technology adoption, and data and AI maturity.
At Compare the Market and iSelect, we’re all about helping people make smart choices – and joining us might just be yours. We offer an environment where your ideas are heard, your contributions are valued, and your growth genuinely matters. Collaboration is part of our DNA and you’ll be joining a supportive, forward-thinking team where you can do meaningful work (and be recognised for it!)
We’re guided by three shared values that shape how we work, how we treat each other, and how we show up for our customers - Drive, Noble Intent and All Together.
Plus, we offer a range of great perks to help you thrive both in and out of work:
- Vibrant and social community who have annual celebrations, family fun days and regular events
- Enjoy flexible work arrangements, including the option of one day working from home each week or a 9-day fortnight
- Enjoy additional leave days - ‘ME’ leave and ‘Volunteer Day’ leave
- Free coffee on Mondays
- Option to purchase additional leave
- Return To Work bonus for returning parents
- Quarterly Award and Recognition program
- Minimum a bachelor’s degree qualified in Computer Science, Analytics, IT or a relevant discipline
- 10+ years' experience in Data Engineering, Analytics, Software Engineering, or Business Intelligence environments, including leading of enterprise-scale data initiatives
- Extensive experience designing, building, and governing cloud-based data platforms across AWS, Azure, or GCP, including data warehouses, lakehouses, and modern data ecosystems
- Strong expertise in data architecture and platform design, including Data Warehouses, Data Lakes, Lakehouses, Data Mesh, Digital win, Medallion Architecture, or scalable enterprise data products.
- Advanced experience with modern data platforms and technologies, including Databricks, Snowflake, Microsoft Fabric, Synapse, Redshift, BigQuery, Spark, streaming technologies, and cloud-native data services.
- Proven capability implementing engineering excellence practices, including DataOps, SecOps, CI/CD, IaC, DevOps, automated testing, observability, platform automation, and operational resilience.
- Deep understanding of enterprise data governance and management disciplines, including data quality, metadata, lineage, cataloguing, master data, security, privacy, access controls, audit, and regulatory compliance
- Experience leading data integration, automation, and platform modernisation initiatives, including APIs, event-driven architectures, reusable frameworks, orchestration platforms, and enterprise application integration
- Strong knowledge of designing infrastructure for AI eccentric solutions on EDWs, including Generative AI, RAG, vector search, embeddings, AI agents, intelligent automation, MLOps, and AI governance principles
- Demonstrated experience defining technology roadmaps, engineering standards, architecture patterns, up-and-coming AI models on the market and strategic platform investments that improve scalability, reliability, security, performance, and cost efficiency
- Proven leadership, stakeholder management, and mentoring capability, with experience developing engineering teams, providing technical direction, influencing strategic outcomes, and driving continuous improvement across Data & Analytics functions ad BAU initiatives.
At Compare the Market and iSelect we’re proud to be an Equal Opportunity Employer and we embrace diversity.
This role is available to candidates with unlimited Australian working rights only. Sponsorship is not available.
iSelect and Compare the Market, brands of iCompare Holdings, are committed to protecting your privacy. All applications and personal information are kept confidential and securely stored in line with Australian privacy laws, used only for recruitment purposes, and not shared with third parties outside iCompare Holdings without your consent.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
