Lead Data Platform Engineer — Databricks on AWS

Cox Purtell; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

Cox Purtell is seeking a Technical Lead / Principal Engineer to own and build end-to-end Databricks Lakehouse platforms on AWS. The role spans architecture, batch & streaming pipelines, and cost-aware tuning, with mentoring and cross-team collaboration in a large data and AI practice.

You will be hands-on across the stack, shaping technical direction while coaching engineers. On-site Sydney role with strong vendor partnerships and scale-focused delivery.

Qualifications

• Proven experience building data platforms end to end, infrastructure through to data lake, not one layer in isolation.
• Deep hands-on Databricks and Delta Lake, including medallion architecture and PySpark or Scala, with AWS (Glue or Redshift at minimum).
• Advanced SQL and performance tuning across distributed systems, with real Spark and cloud troubleshooting experience.
• Databricks certifications (Certified Data Engineer / Certified Professional) and AWS certifications highly regarded, not mandatory.
• Technical Lead level: strong individual delivery on complex builds. Prior leadership experience is valued but not required, clear potential to lead is enough.
• Principal Engineer / Hands-on Architect level: demonstrated ownership of technical direction and coaching engineers through delivery, not just managing tickets.
• Comfortable in genuine ambiguity on large, complex programs, not tightly scoped analytics or reporting projects.
• Full Australian working rights required. We are unable to progress applications without them.

Responsibilities

• Architect end-to-end Databricks Lakehouse platforms, including medallion architecture, on AWS (Glue or Redshift).
• Design and build batch, streaming and real-time pipelines feeding AI and advanced analytics workloads.
• Own distributed systems performance: Spark tuning, query optimisation, and cost and cluster right-sizing at scale.
• Improve CI/CD, infrastructure as code (Terraform) and automated deployment practice across the platform.
• Technical Lead level: own delivery of your components directly, whether you’re already leading others or ready to step into that for the first time.
• Principal Engineer / Hands-on Architect level: set technical direction across the build and coach engineers through the complexity, not just review their output.
• Get direct access to advanced vendor training and close working relationships with the platform partners behind the stack.

Skills

Databricks

Delta Lake

Apache Spark

PySpark

Scala

Medallion Architecture

AWS

Glue

Redshift

Terraform

CI/CD

Data Lake

Job description

Cox Purtell is seeking a Technical Lead / Principal Engineer to own and build end-to-end Databricks Lakehouse platforms on AWS. The role spans architecture, batch & streaming pipelines, and cost-aware tuning, with mentoring and cross-team collaboration in a large data and AI practice.

You will be hands-on across the stack, shaping technical direction while coaching engineers. On-site Sydney role with strong vendor partnerships and scale-focused delivery.