Databricks Tech Lead – Sydney

Delivery Centric Pty Ltd; Sydney; On-site; AUD 180,000 - 260,000; Full time

Job summary

Delivery Centric Pty Ltd is seeking a seasoned Databricks Tech Lead to design, build, and scale data ingestion and processing frameworks. You will lead a team of data engineers, drive software engineering best practices, and develop reusable, scalable pipelines to ingest and transform data from diverse sources into a Lakehouse platform.

The role requires 8+ years in Data Engineering, with 3–4 years in technical leadership, and deep hands-on expertise in PySpark, Databricks SQL, Delta Lake, and

Qualifications

• 8+ years of Data Engineering experience with 3–4 years in technical leadership managing teams or pods.
• Advanced proficiency in PySpark and Databricks SQL with hands‑on Delta Lake and Unity Catalog experience.
• Experience building configurable data ingestion frameworks for structured and unstructured data.
• Strong experience with cloud services (AWS/Azure/GCP) and data modelling concepts such as star/snowflake schemas and SCDs.

Responsibilities

• Design and build metadata-driven ingestion frameworks supporting batch, streaming, and CDC.
• Define architectural standards for Delta Lake Bronze/Silver/Gold with Medallion architecture.
• Optimize Spark jobs for performance, cost, and scalability on Databricks.
• Implement CI/CD pipelines and automated testing following DataOps practices.
• Integrate pipelines with Unity Catalog for governance and lineage.

Skills

PySpark

Databricks SQL

Delta Lake

Unity Catalog

Cloud services (AWS/Azure/GCP)

Data modelling (star/snowflake, SCD)

CI/CD

Git

DataOps

Stakeholder management

Tools

Databricks

Spark

SQL

Job description

Delivery Centric is seeking a seasoned Databricks Tech Lead to design, build, and scale data ingestion and processing frameworks. The role will lead a team of data engineers, drive software engineering best practices, and developreusable, highly scalable pipelines to ingest and transform data from diverse enterprise sources into a Lakehouse platform.

Key Responsibilities

• Design and build metadata-driven, generic data ingestion frameworks supporting batch, streaming, and CDC to automate the onboarding of new data sources.
• Define architectural standards for Medallion architecture (Bronze, Silver, Gold) using Delta Lake.
• Optimise Spark jobs for performance, scalability, and cost efficiency across the Databricks platform.
• Implement and mature CI/CD pipelines using Git and automated testing, while enforcing DataOps best practices.
• Integrate data pipelines with Unity Catalog to support enterprise data governance and lineage.

Required Qualifications

• 8+ years of Data Engineering experience, including 3–4 years of hands‑on technical leadership experience leading engineering teams or pods.
• Advanced proficiency in PySpark and Databricks SQL, with strong hands‑on experience in Delta Lake and Unity Catalog.
• Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
• Strong experience with AWS, Azure, or GCP cloud‑native services such as S3 or ADLS.
• Deep understanding of data modelling, including star/snowflake schemas and SCD (Slowly Changing Dimensions), along with strong software engineering principles applied to data platforms.
• Excellent stakeholder management skills and the ability to translate complex technical architectures into business value.

Preferred Qualifications

• Experience with Kafka, dbt, or Delta Live Tables (DLT).
• Databricks Certified Data Engineer Associate or Professional certification.