Senior Data Engineer

Talent Insights Group; Sydney; Ibrido; AUD 150.000 - 210.000; Tempo pieno

Descrizione del lavoro

Talent Insights Group in Sydney is seeking a Senior Data Engineer to own the curated data pipeline layer in GCP, transforming raw data into scalable pipelines and semantic layers.

You will modernise legacy environments, build modular data products, and support enterprise dashboards, analytics and production ML, through cross‑functional collaboration. This is a hybrid role with an initial 12‑month contract and scope to extend.

Competenze

• 5+ years of commercial Data Engineering or Software Engineering experience.
• Strong Python, SQL and Linux skills.
• Extensive experience with cloud data platforms (GCP BigQuery) or comparable (Snowflake, Databricks).
• Experience with dbt Core and modular data modelling.
• Production workflows with Airflow/Cloud Composer.
• Experience modernising data warehouses and migrations.
• Familiarity with AI tools like Claude Code or Copilot is a plus.

Mansioni

• Architect scalable data pipelines using BigQuery, Airflow, and Vertex AI Pipelines.
• Modernise legacy data foundations into central, governed architectures.
• Build curated data products and semantic layers.
• Apply AI augmentation, testing frameworks, data contracts, and schema drift monitoring.
• Collaborate with platform teams, data scientists, BI analysts, and stakeholders.
• Provide post-deployment support and maintain technical documentation.

Conoscenze

Python

SQL

Linux

Data Modeling

Stakeholder Management

Communication

Strumenti

BigQuery

Airflow / Cloud Composer

Vertex AI Pipelines

dbt Core

Git

Docker

CI/CD

Cloud Platform

Descrizione del lavoro

Sydney | Hybrid

Initial 12 months contract, scope to extend

• Work on large-scale enterprise data modernisation and AI initiatives.
• Build scalable data pipelines and curated data products within a modern cloud environment.
• Opportunity to make a significant impact on enterprise data architecture and engineering practices.

The Opportunity

We are seeking an experienced Senior Data Engineer to join a high-performing engineering team working across large-scale data and AI initiatives.

You will take ownership of the curated data pipeline layer within a Google Cloud Platform (GCP) environment, working with raw data and event streams to build robust, scalable transformation pipelines, semantic layers and curated data products.

These data foundations will support operational dashboards, enterprise reporting, advanced analytics and production machine learning models.

A key focus of the role will be modernising legacy data environments, transitioning siloed data assets into automated, scalable architectures and establishing the foundations for emerging Generative AI capabilities.

Key Responsibilities

• Architect Scalable Data Pipelines: Design, build and optimise high-throughput data transformation pipelines using BigQuery, Cloud Composer / Airflow and Vertex AI Pipelines.
• Modernise Legacy Data Foundations: Refactor and migrate legacy cloud data projects into modern, centrally governed architectures while maintaining downstream stability.
• Build Curated Data Products: Develop modular datasets, domain-oriented data products and semantic layers using dbt Core or similar frameworks.
• AI-Augmented Engineering: Leverage modern AI coding assistants and implement automated testing frameworks, data contracts, unit testing and schema drift monitoring.
• Cross-Functional Collaboration: Work closely with platform teams, data scientists, ML engineers, BI analysts and business stakeholders to deliver scalable data solutions.
• Operational Excellence: Lead post-deployment support, ensure system reliability and provide effective technical documentation and handovers.

What We're Looking For

• 5+ years of commercial Data Engineering or Software Engineering experience.
• Strong hands-on Python, SQL and Linux skills.
• Extensive experience with cloud data platforms, ideally GCP and BigQuery, or equivalent platforms such as Snowflake or Databricks.
• Strong experience with dbt Core or similar data transformation frameworks, including modular data modelling, macros and data contracts.
• Experience building and managing production workflows using Apache Airflow, Cloud Composer or equivalent orchestration tools.
• Experience modernising legacy data warehouse environments and managing complex schema migrations.
• Familiarity with AI developer tools such as Claude Code and GitHub Copilot to accelerate development, refactoring and testing.
• Strong understanding of Git, CI/CD pipelines, GitHub Actions, Docker and automated data quality testing.
• Excellent stakeholder management and communication skills, with the ability to manage competing priorities.

Highly Desirable

• Experience preparing governed data structures, semantic layers and APIs for Generative AI applications and automated analytics agents.
• Exposure to event-driven data streaming technologies such as Google Pub/Sub, Kafka or Dataflow.
• Familiarity with enterprise data governance and cataloguing tools such as Collibra.
• Experience working within retail, supply chain or e-commerce environments.

If you're a Senior Data Engineer who enjoys solving complex data engineering challenges, modernising legacy platforms and working at the intersection of data and AI, we'd love to hear from you.