Data Lead

FR Consultancy; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

FR Consultancy in Sydney, Australia seeks an accomplished Lead Data Engineer to architect and deliver scalable data ingestion platforms. You will guide engineering pods, optimize Spark-based pipelines on Databricks, and shape a Delta Lake-based Medallion architecture for enterprise-grade data.

Strong leadership and stakeholder skills are essential, with a visa status permitting full working rights. Expect to own framework design, mentor junior staff, and drive CI/CD maturity across data

Qualifications

• 8+ years in Data Engineering with 3–4 years of hands-on technical leadership.
• Advanced PySpark and Databricks SQL proficiency.
• Proven experience building configurable data ingestion frameworks.
• Strong cloud experience (AWS, Azure or GCP) with services like S3/ADLS.
• Deep data modeling knowledge (star/snowflake) and SCD practices.
• Excellent stakeholder management and ability to translate architecture to business value.

Responsibilities

• Framework Development: build metadata-driven, generic data ingestion frameworks.
• Architecture & Design: set standards for Medallion architecture using Delta Lake.
• Technical Leadership: mentor engineers; lead design reviews.
• Pipeline Optimization: improve Spark jobs for performance and cost.
• CI/CD & DevOps: mature pipelines and enforce DataOps practices.
• Governance & Security: integrate pipelines with Unity Catalog for governance.

Skills

Data engineering

Technical leadership

PySpark

Databricks SQL

Ingestion frameworks

Cloud platforms (AWS/Azure/GCP)

Data modeling

SCD (Slowly Changing Dimensions)

Stakeholder management

Code reviews

Tools

Databricks

Delta Lake

Kafka

dbt

Delta Live Tables

Git

Job description

Status: Valid visa with full working rights

Key Responsibilities

• Framework Development: Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
• Architecture & Design: Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
• Technical Leadership: Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
• Pipeline Optimization: Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
• CI/CD & DevOps: Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
• Governance & Security: Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.

Required Qualifications

• Experience: 8+ years in Data Engineering, with at least 3–4 years of hands-on technical leadership experience leading engineering pods.
• Core Databricks & Spark: Advanced proficiency in PySpark and Databricks SQL.
• Ingestion Expertise: Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
• Cloud Platforms: Strong experience working with AWS, Azure, or GCP cloud-native services (e.g., S3, ADLS).
• Engineering Standards: Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context.
• Soft Skills: Excellent stakeholder management and the ability to translate complex technical architectures into business value.

Preferred Qualifications

• Experience with Kafka, dbt, or Delta Live Tables (DLT).
• Databricks Certified Data Engineer Associate/Professional credentials

To read about our privacy policy https://www.frconsultancy.com.au/privacy-policy