Databricks Solution Lead - NCS Australia

OpenTalent; Adelaide; On-site; AUD 140,000 - 190,000; Full time

Job summary

OpenTalent seeks a Lead Databricks Engineer to lead design and implementation of Lakehouse platforms on Databricks for enterprise clients.

You will mentor a team, drive best practices in CI/CD and DevOps, and collaborate with stakeholders to deliver scalable data solutions across Azure and AWS ecosystems.

Qualifications

• Strong experience in data engineering (enterprise-scale environments preferred).
• Deep expertise in Databricks, Apache Spark (PySpark preferred), and Delta Lake.
• Proven experience designing and implementing Lakehouse architectures (Medallion model).
• Advanced proficiency in SQL and Python.
• Strong experience with cloud platforms – Azure (ADLS, ADF, Synapse) or AWS (S3, Glue, Redshift).
• Experience building batch and real-time/streaming data pipelines.
• Hands-on experience with data orchestration tools (ADF, Airflow, Databricks Workflows).
• Experience implementing CI/CD pipelines and DevOps practices (Azure DevOps, GitHub, or similar).
• Working knowledge of Infrastructure-as-Code (Terraform or similar tools).
• Strong understanding of data modelling and data warehousing concepts.
• Experience working in client-facing consulting or system integrator environments.
• Proven ability to lead and mentor engineering teams, including onshore/offshore models.
• Strong stakeholder management, communication, and presentation skills.

Responsibilities

• Lead design and implementation of scalable data platforms on Databricks for enterprise clients.
• Mentor and guide a team of data engineers, promoting best practices in CI/CD and DevOps.
• Collaborate with architects and stakeholders to translate business needs into data solutions.
• Build and optimize batch and real-time data pipelines, dashboards, and ingestion frameworks.
• Ensure data governance, security, privacy, metadata, and lineage using Unity Catalog and related tools.

Skills

Leadership

Client-facing

Stakeholder management

Mentoring

Communication

Delivery ownership

Cloud familiarity

Tools

Databricks

Apache Spark

Delta Lake

SQL

Python

Azure Data Factory

Azure Synapse

AWS Glue

S3

Snowflake

Redshift

Airflow

Databricks Workflows

Terraform

Azure DevOps

GitHub

Infrastructure as Code

Unity Catalog

Structured Streaming

Delta Live Tables

Job description

As a Lead Databricks Engineer, you will play a critical role in delivering large-scale data transformation programs. You will lead the design and implementation of modern Lakehouse platforms on Databricks, working closely with client stakeholders, architects, and cross-functional teams. This is a client-facing leadership role requiring strong consulting capabilities, technical depth, and delivery ownership.

The Lead Databricks Engineer is responsible for designing and implementing scalable data platforms using Databricks, Apache Spark, and Delta Lake, including building robust batch and real-time data pipelines and developing ingestion frameworks for both structured and unstructured data while optimising performance, cost, and reliability. The role involves leading and mentoring a team of data engineers, driving best practices across coding, CI/CD, and DevOps, and collaborating with cross-functional stakeholders to translate business needs into effective data solutions, while owning end-to-end project delivery. Additionally, the engineer will implement data solutions across cloud platforms—primarily Azure, but also AWS or GCP—integrating with services such as Azure Data Factory, AWS Glue, and modern data warehouses like Snowflake or Synapse, while managing infrastructure through Infrastructure-as-Code tools. A strong focus is placed on data governance, including implementing quality checks, monitoring and observability frameworks, ensuring compliance with security and privacy standards, and managing metadata and lineage using tools like Unity Catalog. The role also supports advanced analytics by enabling data science workflows, building ML-ready datasets using Databricks capabilities, and supporting real-time analytics through technologies such as Structured Streaming and Delta Live Tables.

• Strong experience in data engineering (enterprise-scale environments preferred).
• Deep expertise in Databricks, Apache Spark (PySpark preferred), and Delta Lake.
• Proven experience designing and implementing Lakehouse architectures (Medallion model).
• Advanced proficiency in SQL and Python.
• Strong experience with cloud platforms – Azure (ADLS, ADF, Synapse) or AWS (S3, Glue, Redshift).
• Experience building batch and real-time/streaming data pipelines.
• Hands-on experience with data orchestration tools (ADF, Airflow, Databricks Workflows).
• Experience implementing CI/CD pipelines and DevOps practices (Azure DevOps, GitHub, or similar).
• Working knowledge of Infrastructure-as-Code (Terraform or similar tools).
• Strong understanding of data modelling and data warehousing concepts.
• Experience working in client-facing consulting or system integrator environments.
• Proven ability to lead and mentor engineering teams, including onshore/offshore models.
• Strong stakeholder management, communication, and presentation skills.