Lead Data Engineer

Indigitise Pty Ltd; Canberra; On-site; AUD 140,000 - 180,000; Full time

Job summary

Indigitise Pty Ltd is seeking a Data Engineer to join the Enterprise Analytics and Technology Services team to work across data platforms and analytics.

The role focuses on ETL/ELT development, data movement, and the creation of data visualisations using Power BI, with Azure Databricks and Data Factory as core technologies in a modern cloud environment. You will contribute to design, development, and testing within an Agile team.

Qualifications

• Minimum 5 years' experience as a data engineer.
• Experience with Azure cloud technologies and Databricks.
• Experience building ETL/ELT pipelines and data products.
• Ability to design, test, and deploy data movement and transformation solutions.
• Strong communication and collaboration in Agile teams.

Responsibilities

• Design, develop, and test data movement and transformation processes.
• Create and maintain automated ingest and transformation patterns.
• Collaborate with project teams to meet departmental priorities.
• Support data engineers in delivery teams with quality reviews.
• Develop data visualisations and dashboards using Power BI.

Skills

ETL/ELT development

Agile methodologies

Leadership / ownership

Data movement & transformation

Communication skills

Education

Relevant data engineering qualifications or training

Tools

Azure Databricks

Azure Data Factory

SQL Server

Data Lake Storage

Power BI

DevOps

Visual Studio

Oracle

Job description

Our federal client is looking for a Data Engineer to join its Enterprise Analytics and Technology Services (EATS) section to work across its data and analytics platforms. We are seeking candidates with strong experience in developing Extract, Transform, Load (ETL) and Extract, Load, Transform (ELT) processes, and /or the development of data products and complex data visualisations.

The role will be responsible for design, development and testing activities across several data movement, data transformation, and data visualisation processes within DAFF. The data movement and transformation processes focus on the preparation of data for use in decision making processes across the department, utilising modern cloud technology (Azure & Databricks) to enable operational analytics use cases.

To be successful in the role, you must have a strong work ethic including taking ownership, providing leadership, having a high level of productivity, and working with a multi-disciplinary team in an Agile development environment. You must have suitable qualifications or training in data engineering techniques and at least 5 years’ experience working as a data engineer.

Requirements

Responsibilities

include, but not limited to:

• Be responsive, flexible, and work collaboratively as part of an agile team.
• Strong relationship building, and negotiation skills.
• Strong written and oral communication skills.
• Creating and maintaining automated ingest and transformation patterns and frameworks.
• Designing, building and maintaining data ingest and transformation solutions to meet current and emerging needs.
• Assisting project teams achieve objectives that align with departmental, divisional, and program priorities.
• Supporting data engineers in delivery teams, through regular quality reviews and constructive feedback on utilising data assets to produce quality data products.

The successful candidate will require experience with the following techniques and technologies:

• Azure hosted services including:
  • Data Integration - Data Factory, SQL Server Integration Services and/or Databricks
  • Data Store - SQL Server and/or Data Lake Storage
  • Analytics - Azure Databricks, Azure Machine Learning, ArcGIS Enterprise
• Development tools – DevOps, Visual Studio
• Data technology solutions – sourcing (Oracle, Ingres, Azure, SQL Server), automated ingestion
• Data Preparation
  • Transformation of data into formats tailored for advanced analytics and AI use cases – Parquet and/or Delta
• Data Visualisation
  • Analyse and interpret complex data sets, and to identify trends and patterns.
  • Design principles to create appropriate visualisations for target audience.
  • Visualisation tools - Power BI.