Senior Technical Lead

HCL Australia Services PTY. Limited; Australia; On-site; AUD 180,000 - 250,000; Full time

Job summary

HCL Australia Services PTY. Limited is seeking a Senior Technical Lead to design, build, and scale our data ingestion and processing framework on the Databricks platform.

You will lead a team of data engineers, promote best practices, and deliver reusable, scalable pipelines ingesting data from diverse sources into our Lakehouse. You will own framework development, architecture, and governance, mentoring engineers and driving code quality, performance, and security across cloud-native

Qualifications

• 8+ years in Data Engineering with leadership experience
• Advanced proficiency in PySpark and Databricks SQL
• Experience building data ingestion frameworks for structured and unstructured data
• Experience with cloud-native services (AWS/Azure/GCP)

Responsibilities

• Design and build metadata-driven ingestion frameworks (batch, streaming, CDC) to onboard new sources
• Define architectural standards for the Delta Lake Medallion architecture (Bronze, Silver, Gold)
• Lead technical design discussions, mentor engineers, and conduct code reviews
• Optimize Spark jobs for performance, scalability, and cost on the Databricks platform
• Implement and mature CI/CD pipelines for data platforms and enforce DataOps practices
• Integrate ingestion pipelines with Unity Catalog for governance and lineage

Skills

Databricks

PySpark

Spark SQL

AWS

Azure

GCP

Architecture design

Leadership

Tools

Delta Lake

Unity Catalog

Git

Job description

Career Opportunities: Senior Technical Lead (168964)

Requisition ID168964-Posted09/30/2026

We are seeking a seasoned Databricks Tech Leadto design, build, and scale our next-generation data ingestion and processing framework. In this role, you will lead a team of data engineers, champion best practices in software development, and build reusable, highly scalable pipelines to ingest and transform data from diverse enterprise sources into our Lakehouse platform.

Key Responsibilities

• Framework Development:Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
• Architecture & Design:Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
• Technical Leadership:Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
• Pipeline Optimization:Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
• CI/CD & DevOps:Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
• Governance & Security:Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.

Other Requirements

• Experience:8+ years in Data Engineering, with at least 3–4 years of hands-on technical leadership experience leading engineering pods.
• Core Databricks & Spark:Advanced proficiency in PySpark and Databricks SQL.
• Ingestion Expertise:Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
• Cloud Platforms:Strong experience working with AWS, Azure, or GCP cloud-native services (e.g., S3, ADLS).
• Engineering Standards:Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context.
• Soft Skills:Excellent stakeholder management and the ability to translate complex technical architectures into business value.