Databricks Tech Lead — Data Ingestion & Lakehouse Architect

HCL Australia Services PTY. Limited; Australia; On-site; AUD 180,000 - 250,000; Full time

Job summary

HCL Australia Services PTY. Limited is seeking a Senior Technical Lead to design, build, and scale our data ingestion and processing framework on the Databricks platform.

You will lead a team of data engineers, promote best practices, and deliver reusable, scalable pipelines ingesting data from diverse sources into our Lakehouse. You will own framework development, architecture, and governance, mentoring engineers and driving code quality, performance, and security across cloud-native

Qualifications

• 8+ years in Data Engineering with leadership experience
• Advanced proficiency in PySpark and Databricks SQL
• Experience building data ingestion frameworks for structured and unstructured data
• Experience with cloud-native services (AWS/Azure/GCP)

Responsibilities

• Design and build metadata-driven ingestion frameworks (batch, streaming, CDC) to onboard new sources
• Define architectural standards for the Delta Lake Medallion architecture (Bronze, Silver, Gold)
• Lead technical design discussions, mentor engineers, and conduct code reviews
• Optimize Spark jobs for performance, scalability, and cost on the Databricks platform
• Implement and mature CI/CD pipelines for data platforms and enforce DataOps practices
• Integrate ingestion pipelines with Unity Catalog for governance and lineage

Skills

Databricks

PySpark

Spark SQL

AWS

Azure

GCP

Architecture design

Leadership

Tools

Delta Lake

Unity Catalog

Git

Job description

HCL Australia Services PTY. Limited is seeking a Senior Technical Lead to design, build, and scale our data ingestion and processing framework on the Databricks platform.

You will lead a team of data engineers, promote best practices, and deliver reusable, scalable pipelines ingesting data from diverse sources into our Lakehouse. You will own framework development, architecture, and governance, mentoring engineers and driving code quality, performance, and security across cloud-native