Lead Data Engineer: Pipelines, Modeling & AI Insights

TT Engineering; Sydney; Hybrid; AUD 150,000 - 190,000; Full time

Benefits offered by this job

Flexible hybrid working

Global career pathways

Continuous learning

Wellbeing & lifestyle allowance

Loyalty leave

Job summary

Endava is seeking a Lead Data Engineer to design, build and operate reliable data platforms that convert complex data into actionable business insights. You’ll work across the data lifecycle, crafting pipelines, ontologies and semantic models while guiding a team of engineers.

You’ll collaborate with clients, analysts and data scientists, implementing RBAC, CI/CD and DevOps practices to ensure scalable, secure data solutions and impactful visualisations for leadership and stakeholders.

Qualifications

• Hands‑on experience delivering production data solutions using one or more of Databricks, Snowflake or Microsoft Fabric.
• Strong programming skills in Python and SQL.
• Experience designing, developing and operating data pipelines, including ETL/ELT processes, orchestration and monitoring.
• A sound understanding of data architecture and modelling techniques, including analytical and dimensional modelling.
• An understanding of ontologies and semantic modelling, and how they connect data structures to business concepts.
• Practical experience optimising data processing and troubleshooting pipeline, query and platform performance.
• Experience integrating data through APIs, including authentication, error handling and reliable data exchange.
• Experience applying role‑based access control (RBAC) and data security best practices.
• Experience using Git, maintaining repositories, writing automated tests and building CI/CD workflows for data solutions.
• Experience developing data visualisations and reports.
• The ability to explain technical choices clearly, work collaboratively with business and technical stakeholders, and guide other engineers.

Responsibilities

• Work directly with clients to translate business requirements into technical solutions; Providing guidance and contributing to design decisions.
• Design, build and maintain data pipelines to ingest, transform and deliver data from a range of sources, including APIs.
• Develop solutions using Databricks, Snowflake or Microsoft Fabric, selecting approaches that suit the client’s requirements and existing environment.
• Contribute to data architecture and solution design, including data storage, integration and modelling.
• Develop data models, ontologies and semantic layers that establish consistent business definitions and support analytics and reporting.
• Implement pipeline orchestration, scheduling, monitoring and operational support to keep data services reliable.
• Troubleshoot data and pipeline issues, improving performance, scalability and cost efficiency.
• Apply data quality controls, automated testing and validation throughout the data lifecycle.
• Implement role‑based access control and follow security best practices for data platforms and integrations.
• Apply DevOps practices to data engineering, including version control, repository management, CI/CD and automated deployment.
• Build data visualisations and reports that make information clear and useful to business users.
• Document solutions, review code and share knowledge to support maintainable delivery and continuous improvement.

Skills

Python

SQL

Databricks

Snowflake

Microsoft Fabric

Data visualization

RBAC

CI/CD

Git

Tools

Databricks

Snowflake

Microsoft Fabric

Job description

Endava is seeking a Lead Data Engineer to design, build and operate reliable data platforms that convert complex data into actionable business insights. You’ll work across the data lifecycle, crafting pipelines, ontologies and semantic models while guiding a team of engineers.

You’ll collaborate with clients, analysts and data scientists, implementing RBAC, CI/CD and DevOps practices to ensure scalable, secure data solutions and impactful visualisations for leadership and stakeholders.