Data Platform Lead (Spark/Databricks & Ingestion)

FR Consultancy; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

FR Consultancy in Sydney, Australia seeks an accomplished Lead Data Engineer to architect and deliver scalable data ingestion platforms. You will guide engineering pods, optimize Spark-based pipelines on Databricks, and shape a Delta Lake-based Medallion architecture for enterprise-grade data.

Strong leadership and stakeholder skills are essential, with a visa status permitting full working rights. Expect to own framework design, mentor junior staff, and drive CI/CD maturity across data

Qualifications

• 8+ years in Data Engineering with 3–4 years of hands-on technical leadership.
• Advanced PySpark and Databricks SQL proficiency.
• Proven experience building configurable data ingestion frameworks.
• Strong cloud experience (AWS, Azure or GCP) with services like S3/ADLS.
• Deep data modeling knowledge (star/snowflake) and SCD practices.
• Excellent stakeholder management and ability to translate architecture to business value.

Responsibilities

• Framework Development: build metadata-driven, generic data ingestion frameworks.
• Architecture & Design: set standards for Medallion architecture using Delta Lake.
• Technical Leadership: mentor engineers; lead design reviews.
• Pipeline Optimization: improve Spark jobs for performance and cost.
• CI/CD & DevOps: mature pipelines and enforce DataOps practices.
• Governance & Security: integrate pipelines with Unity Catalog for governance.

Skills

Data engineering

Technical leadership

PySpark

Databricks SQL

Ingestion frameworks

Cloud platforms (AWS/Azure/GCP)

Data modeling

SCD (Slowly Changing Dimensions)

Stakeholder management

Code reviews

Tools

Databricks

Delta Lake

Kafka

dbt

Delta Live Tables

Git

Job description

FR Consultancy in Sydney, Australia seeks an accomplished Lead Data Engineer to architect and deliver scalable data ingestion platforms. You will guide engineering pods, optimize Spark-based pipelines on Databricks, and shape a Delta Lake-based Medallion architecture for enterprise-grade data.

Strong leadership and stakeholder skills are essential, with a visa status permitting full working rights. Expect to own framework design, mentor junior staff, and drive CI/CD maturity across data