Databricks Tech Lead: Scale Data Ingestion & Lakehouse

Delivery Centric Pty Ltd; Sydney; On-site; AUD 180,000 - 260,000; Full time

Job summary

Delivery Centric Pty Ltd is seeking a seasoned Databricks Tech Lead to design, build, and scale data ingestion and processing frameworks. You will lead a team of data engineers, drive software engineering best practices, and develop reusable, scalable pipelines to ingest and transform data from diverse sources into a Lakehouse platform.

The role requires 8+ years in Data Engineering, with 3–4 years in technical leadership, and deep hands-on expertise in PySpark, Databricks SQL, Delta Lake, and

Qualifications

• 8+ years of Data Engineering experience with 3–4 years in technical leadership managing teams or pods.
• Advanced proficiency in PySpark and Databricks SQL with hands‑on Delta Lake and Unity Catalog experience.
• Experience building configurable data ingestion frameworks for structured and unstructured data.
• Strong experience with cloud services (AWS/Azure/GCP) and data modelling concepts such as star/snowflake schemas and SCDs.

Responsibilities

• Design and build metadata-driven ingestion frameworks supporting batch, streaming, and CDC.
• Define architectural standards for Delta Lake Bronze/Silver/Gold with Medallion architecture.
• Optimize Spark jobs for performance, cost, and scalability on Databricks.
• Implement CI/CD pipelines and automated testing following DataOps practices.
• Integrate pipelines with Unity Catalog for governance and lineage.

Skills

PySpark

Databricks SQL

Delta Lake

Unity Catalog

Cloud services (AWS/Azure/GCP)

Data modelling (star/snowflake, SCD)

CI/CD

Git

DataOps

Stakeholder management

Tools

Databricks

Spark

SQL

Job description

Delivery Centric Pty Ltd is seeking a seasoned Databricks Tech Lead to design, build, and scale data ingestion and processing frameworks. You will lead a team of data engineers, drive software engineering best practices, and develop reusable, scalable pipelines to ingest and transform data from diverse sources into a Lakehouse platform.

The role requires 8+ years in Data Engineering, with 3–4 years in technical leadership, and deep hands-on expertise in PySpark, Databricks SQL, Delta Lake, and