Lead Data Engineer

XPT Software; City of Melbourne; On-site; AUD 180,000 - 240,000; Full time

Job summary

XPT Software in Melbourne, Australia seeks a Lead Data Engineer to drive the strategy, architecture, delivery, and optimization of enterprise-scale data platforms on Microsoft Azure. You will lead data platform roadmap, govern pipelines, models, and lakehouse solutions, and mentor teams.

The role requires 10+ years in IT and 5+ years leadership in data engineering, with strong expertise in SQL, Python, PySpark, and Azure services.

Qualifications

• Strong expertise in SQL, Python, and PySpark.
• Extensive experience with Azure Data Factory (ADF) and Azure Synapse Analytics.
• Hands-on Delta Lake and lakehouse implementations.
• Advanced data modeling and data warehousing concepts.
• Experience with the Microsoft Fabric ecosystem.
• Proven cloud-native data architectures for enterprise.
• Strong design thinking and problem-solving abilities.
• Experience with CI/CD pipelines and DevOps practices.
• Experience in cloud cost optimization, performance tuning, and resource governance.

Responsibilities

• Lead end-to-end delivery of enterprise data platforms, including architecture, development, deployment, and optimization.
• Define and drive the data platform roadmap aligned with business objectives and strategic priorities.
• Architect and govern scalable data pipelines, data models, data warehouses, and lakehouse solutions using Azure services.
• Provide technical leadership to Data Engineering teams through design reviews, code reviews, and best-practice implementation.
• Drive adoption of Microsoft Fabric, APIs, Azure Functions, and Python-based solutions.
• Ensure platform scalability, reliability, security, and cost optimization while maintaining governance.
• Establish and enforce data governance, data quality, security, and compliance standards.
• Implement CI/CD, DevOps, and Agile delivery practices across the data engineering landscape.
• Collaborate with stakeholders to translate requirements into scalable technical solutions.
• Mentor and develop Data Engineering talent, fostering learning and innovation.
• Leverage AI-assisted tooling to improve productivity and solution quality.
• Drive modernization by evaluating emerging technologies.

Skills

SQL

Python

PySpark

Azure Data Factory

Azure Synapse Analytics

Delta Lake

Data Modeling

Data Warehousing

Microsoft Fabric

Cloud-native architectures

Design thinking

CI/CD / DevOps

Cloud cost optimization

Performance tuning

Resource governance

GitHub Copilot

AI-assisted engineering

APIs

Tools

Power BI

Azure Functions

GitHub Copilot

Job description

Mandatory Skill Set: SQL + Python + PySpark + Azure Synapse

Role Overview

We are seeking a highly experienced Lead Data Engineer to drive the strategy, architecture, delivery, and optimization of enterprise-scale data platforms on Microsoft Azure. The ideal candidate will combine deep technical expertise with strong leadership, stakeholder management, and delivery governance capabilities to build scalable, secure, and cost-efficient data solutions.

Must-Have Skills

• Strong expertise in SQL, Python, and PySpark
• Extensive experience with Azure Data Factory (ADF) and Azure Synapse Analytics.
• Hands-on experience with Delta Lake architecture and modern data lakehouse implementations.
• Advanced knowledge of Data Modeling and Data Warehousing concepts.
• Strong exposure to the Microsoft Fabric ecosystem.
• Proven expertise in designing cloud-native data architectures and enterprise-scale solutions.
• Strong design thinking and problem-solving capabilities.
• Working knowledge of CI/CD pipelines and DevOps practices.
• Experience in cloud cost optimization, performance tuning, and resource governance.

Key Responsibilities

• Lead and own the end-to-end delivery of enterprise data platforms, including architecture, development, deployment, and optimization.
• Define and drive the data platform roadmap, ensuring alignment with business objectives and strategic priorities.
• Architect and govern scalable data pipelines, data models, data warehouses, and lakehouse solutions using Azure services.
• Provide technical leadership to Data Engineering teams through design reviews, code reviews, and implementation of engineering best practices.
• Drive the adoption and implementation of Microsoft Fabric, APIs, Azure Functions, and Python-based solutions.
• Ensure platform scalability, reliability, security, and operational excellence while optimizing cloud costs.
• Establish and enforce data governance, data quality, security, and compliance standards.
• Implement and drive CI/CD, DevOps, and Agile delivery practices across the data engineering landscape.
• Collaborate with business stakeholders, architects, and technology teams to translate business requirements into scalable technical solutions.
• Mentor, coach, and develop Data Engineering talent, fostering a culture of continuous learning and innovation.
• Leverage GitHub Copilot and AI-assisted engineering tools to improve productivity and solution quality.
• Drive innovation by evaluating emerging technologies and identifying opportunities for platform modernization.

Experience

• 10+ years of overall IT experience, including 10+ years in Data Engineering.
• Minimum 5+ years of leadership experience managing large-scale Data Engineering programs and teams.
• Proven track record of leading cross-functional teams and delivering complex enterprise data transformation initiatives.
• Experience building, hiring, and scaling high-performing Data Engineering teams.
• Experience working within multi-vendor and globally distributed delivery environments.
• Strong experience in stakeholder management, delivery governance, and executive-level communication.

Preferred Skills

• Experience building and managing enterprise APIs and Azure Function Apps.
• Exposure to framework-based data engineering development approaches.
• Experience in solution architecture, stakeholder engagement, and delivery governance.
• Azure certifications such as:
• Azure Solutions Architect Expert
• Familiarity with MLOps, Data Science, and AI-driven analytics solutions.
• Hands-on experience with Power BI and enterprise reporting platforms.
• Experience leveraging AI and Generative AI capabilities within Data Engineering ecosystems.
• Strong ownership mindset with a focus on accountability and business outcomes.
• Excellent communication, stakeholder management, and influencing skills.
• Ability to balance technical leadership with delivery and people management responsibilities.
• Strategic thinker with a strong focus on execution excellence.
• Strong conflict resolution and cross-functional collaboration skills.
• Adaptability and resilience in fast-paced, evolving environments.
• Passion for mentoring teams and fostering a culture of innovation and continuous improvement.