Senior Data Engineer: Databricks & Spark Lead

Full Circle Partners; Western Australia; On-site; AUD 165,000 - 223,000; Full time

Job summary

Full Circle Partners in Perth invites an experienced Senior Data Engineer to join a major data transformation program on a 12-month contract. You will design and deliver scalable data solutions across the Databricks ecosystem while mentoring an internal data engineering team.

You will work with architects and data teams to establish standards, develop ETL/ELT pipelines, integrate data from APIs, Kafka and databases, and implement CI/CD and governance practices.

Qualifications

• Extensive experience as a Senior Data Engineer with Databricks and Spark.
• Advanced SQL and Python skills for complex data pipelines.
• Experience designing enterprise-scale data platforms and solutions.
• Understanding of modern data engineering patterns, architecture and practices.
• Experience with orchestration and transformation technologies such as Airflow and dbt.
• Experience integrating data from APIs, relational databases and streaming platforms like Kafka.
• Experience implementing CI/CD in data environments.
• Exposure to data catalogue, governance and metadata management.

Responsibilities

• Design, build and maintain scalable ETL/ELT pipelines and enterprise data platform solutions using the Databricks ecosystem.
• Provide hands-on technical mentoring to data engineers, supporting code quality, design patterns and engineering best practices.
• Develop and implement solution designs covering data migration, replication, transformation and integration.
• Build data ingestion pipelines from a range of sources, including SQL databases, APIs and streaming platforms such as Kafka.
• Work with complex data structures, including JSON, and transform data into efficient formats.
• Develop and maintain pipeline orchestration and workflow solutions using modern tools and frameworks.
• Implement CI/CD practices and automated infrastructure to support reliable data platform delivery.
• Contribute to data governance, catalogue management and platform standards.
• Explain and document technical decisions to stakeholders.
• Collaborate with architects, developers and business stakeholders to deliver robust data solutions.

Skills

Databricks

Apache Spark

SQL

Python

Airflow

dbt

Kafka

CI/CD

Data governance

Mentoring

Communication

Tools

Airflow

Job description

Full Circle Partners in Perth invites an experienced Senior Data Engineer to join a major data transformation program on a 12-month contract. You will design and deliver scalable data solutions across the Databricks ecosystem while mentoring an internal data engineering team.

You will work with architects and data teams to establish standards, develop ETL/ELT pipelines, integrate data from APIs, Kafka and databases, and implement CI/CD and governance practices.