Senior Data Engineer / Data Architect - Databricks & AWS

Sociable Tech; Sydney; On-site; AUD 168,000 - 250,000; Full time

Benefits offered by this job

Superannuation

Job summary

Cox Purtell is hiring a Technical Lead / Principal Engineer / Hands-on Architect for Data Engineering (Platform & Infrastructure) in Sydney, NSW. You will own end-to-end Databricks Lakehouse platforms on AWS, delivering medallion architectures, batch and streaming pipelines, with real-world tradeoffs and ownership of the platform.

You'll join a 500+ strong data and AI practice, collaborate with tier-one vendors, and guide engineers through complex builds.

Qualifications

• Proven experience building data platforms end to end, infrastructure through to data lake.
• Deep hands-on Databricks and Delta Lake, including medallion architecture, PySpark or Scala, with AWS (Glue or Redshift).
• Advanced SQL and performance tuning across distributed systems with real Spark and cloud troubleshooting.
• Databricks certifications (Certified Data Engineer / Certified Professional) and AWS certifications highly regarded.

Responsibilities

• Architect end-to-end Databricks Lakehouse platforms on AWS, including medallion architecture and batch/streaming pipelines.
• Own distributed system performance, Spark tuning, cost control, and CI/CD improvements across the platform.
• Set technical direction for the build and coach engineers through delivery with leadership potential.

Skills

Databricks

Delta Lake

Apache Spark

PySpark

Scala

Medallion Architecture

AWS

Glue

Redshift

Terraform

CI/CD

Data Platform

Data Lake

Tools

Databricks

Delta Lake

Apache Spark

PySpark

Scala

Medallion Architecture

AWS

Glue

Redshift

Terraform

CI/CD

Job description

Job Type: Permanent position | Full time position

Job Description

Cox Purtell TECHNICAL LEAD / PRINCIPAL ENGINEER / HANDS-ON ARCHITECT - DATA ENGINEERING (PLATFORM & INFRASTRUCTURE) Sydney, NSW (on-site) | Full Australian working rights required - we're unable to progress candidates without them. You've built a Databricks and AWS data platform from nothing before: medallion architecture, Delta Lake, real pipelines and real tradeoffs, not a diagram someone else handed you.

If that's familiar, keep reading. Sydney-based, Technical Lead through to Hands-on Architect, genuinely hands-on at every level.

You've built a platform from nothing before. This role needs that again. Haven't led a team yet?

If you're ready to, this is where you start. Join a 500+ strong data and AI practice, backed by tier-one vendor alliances.

A top-tier global professional services firm delivering large-scale data and AI transformation programs for major enterprise and government clients across Australia. This is deep, complex platform and infrastructure build work, not light analytics or reporting. You get total ownership of the platforms you build, structured investment in advanced technical training and direct access to the vendors behind the stack, not just a licence and a support ticket.

You're joining a data and AI practice of 500+ specialist practitioners nationally, sitting inside tier-one vendor alliances spanning AWS Databricks, Google Cloud, dbt Labs, Snowflake, Salesforce and Informatica. The programs here run at genuine scale: think large transport-network digital twin builds, national-scale public health rollout automation, and sustainability data initiatives delivered across Australia and the wider region.

Career development is coaching-led, with benefits including dedicated wellbeing leave, paid volunteering and market-leading parental leave. If you've built a data platform from the ground up before, you already know the difference between that and being handed one to extend.

This is the first kind. You'll be designing and building end-to-end Databricks Lakehouse platforms including medallion architecture, batch and streaming pipelines, and Delta Lake at the core, backed by AWS (Glue or Redshift), not a single layer someone else designed around you. How much of that you own depends on where you sit, and leadership experience isn't a prerequisite at every level.

As a Technical Lead, you might already be leading a team, or you might be ready to step into that for the first time with real support behind you. As a Principal Engineer or Hands-on Architect, you're setting technical direction and coaching engineers through the complexity yourself. Either way, you own the platform, not a slice of it.

Duties

• Architect end-to-end Databricks Lakehouse platforms, including medallion architecture, on AWS (Glue or Redshift). Design and build batch, streaming and real-time pipelines feeding AI and advanced analytics workloads.
• Own distributed systems performance: Spark tuning, query optimisation, and cost and cluster right-sizing at scale. Improve CI/CD, infrastructure as code (Terraform) and automated deployment practice across the platform. Technical Lead level: own delivery of your components directly, whether you're already leading others or ready to step into that for the first time.
• Principal Engineer / Hands-on Architect level: set technical direction across the build and coach engineers through the complexity, not just review their output. Get direct access to advanced vendor training and close working relationships with the platform partners behind the stack.

Requirements

• Proven experience building data platforms end to end, infrastructure through to data lake, not one layer in isolation.
• Deep hands-on Databricks and Delta Lake, including medallion architecture and PySpark or Scala, with AWS (Glue or Redshift at minimum).
• Advanced SQL and performance tuning across distributed systems, with real Spark and cloud troubleshooting experience.
• Databricks certifications (Certified Data Engineer / Certified Professional) and AWS certifications highly regarded, not mandatory. Technical Lead level: strong individual delivery on complex builds. Prior leadership experience is valued but not required, clear potential to lead is enough.
• Principal Engineer / Hands-on Architect level: demonstrated ownership of technical direction and coaching engineers through delivery, not just managing tickets. Comfortable in genuine ambiguity on large, complex programs, not tightly scoped analytics or reporting projects. Full Australian working rights required.
• We are unable to progress applications without them.

Skills: Databricks, Delta Lake, Apache Spark, PySpark, Scala, Medallion Architecture, AWS, Glue, Redshift, Terraform, CI/CD, Data Engineering, Data Architecture, Data Platform, Data Lake, Sydney.

Benefits: Superannuation

Work Type: Full time

Salary: $168000-250000