Senior Data Engineer / Data Architect - Databricks & AWS

Cox Purtell; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

Cox Purtell is seeking a Technical Lead / Principal Engineer to own and build end-to-end Databricks Lakehouse platforms on AWS. The role spans architecture, batch & streaming pipelines, and cost-aware tuning, with mentoring and cross-team collaboration in a large data and AI practice.

You will be hands-on across the stack, shaping technical direction while coaching engineers. On-site Sydney role with strong vendor partnerships and scale-focused delivery.

Qualifications

• Proven experience building data platforms end to end, infrastructure through to data lake, not one layer in isolation.
• Deep hands-on Databricks and Delta Lake, including medallion architecture and PySpark or Scala, with AWS (Glue or Redshift at minimum).
• Advanced SQL and performance tuning across distributed systems, with real Spark and cloud troubleshooting experience.
• Databricks certifications (Certified Data Engineer / Certified Professional) and AWS certifications highly regarded, not mandatory.
• Technical Lead level: strong individual delivery on complex builds. Prior leadership experience is valued but not required, clear potential to lead is enough.
• Principal Engineer / Hands-on Architect level: demonstrated ownership of technical direction and coaching engineers through delivery, not just managing tickets.
• Comfortable in genuine ambiguity on large, complex programs, not tightly scoped analytics or reporting projects.
• Full Australian working rights required. We are unable to progress applications without them.

Responsibilities

• Architect end-to-end Databricks Lakehouse platforms, including medallion architecture, on AWS (Glue or Redshift).
• Design and build batch, streaming and real-time pipelines feeding AI and advanced analytics workloads.
• Own distributed systems performance: Spark tuning, query optimisation, and cost and cluster right-sizing at scale.
• Improve CI/CD, infrastructure as code (Terraform) and automated deployment practice across the platform.
• Technical Lead level: own delivery of your components directly, whether you’re already leading others or ready to step into that for the first time.
• Principal Engineer / Hands-on Architect level: set technical direction across the build and coach engineers through the complexity, not just review their output.
• Get direct access to advanced vendor training and close working relationships with the platform partners behind the stack.

Skills

Databricks

Delta Lake

Apache Spark

PySpark

Scala

Medallion Architecture

AWS

Glue

Redshift

Terraform

CI/CD

Data Lake

Job description

TECHNICAL LEAD / PRINCIPAL ENGINEER / HANDS-ON ARCHITECT - DATA ENGINEERING (PLATFORM & INFRASTRUCTURE)

Sydney, NSW (on-site) | Full Australian working rights required - we\'re unable to progress candidates without them.

You’ve built a Databricks and AWS data platform from nothing before: medallion architecture, Delta Lake, real pipelines and real tradeoffs, not a diagram someone else handed you. If that’s familiar, keep reading. Sydney-based, Technical Lead through to Hands-on Architect, genuinely hands-on at every level.

SEEK SELLING POINTS

• You've built a platform from nothing before. This role needs that again.
• Haven't led a team yet? If you\'re ready to, this is where you start.
• Join a 500+ strong data and AI practice, backed by tier-one vendor alliances.

ABOUT THE CLIENT

A top-tier global professional services firm delivering large-scale data and AI transformation programs for major enterprise and government clients across Australia. This is deep, complex platform and infrastructure build work, not light analytics or reporting. You get total ownership of the platforms you build, structured investment in advanced technical training, and direct access to the vendors behind the stack, not just a licence and a support ticket.

ABOUT THE TEAM

You’d be joining a data and AI practice of 500+ specialist practitioners nationally, sitting inside tier-one vendor alliances spanning AWS, Databricks, Google Cloud, dbt Labs, Snowflake, Salesforce and Informatica. The programs here run at genuine scale: think large transport-network digital twin builds, national-scale public health rollout automation, and sustainability data initiatives delivered across Australia and the wider region. Career development is coaching-led, with benefits including dedicated wellbeing leave, paid volunteering and market-leading parental leave.

ABOUT THE ROLE

If you’ve built a data platform from the ground up before, you already know the difference between that and being handed one to extend. This is the first kind. You’ll be designing and building end-to-end Databricks Lakehouse platforms including medallion architecture, batch and streaming pipelines, and Delta Lake at the core, backed by AWS (Glue or Redshift), not a single layer someone else designed around you.

How much of that you own depends on where you sit, and leadership experience isn\'t a prerequisite at every level. As a Technical Lead, you might already be leading a team, or you might be ready to step into that for the first time with real support behind you. As a Principal Engineer or Hands-on Architect, you’re setting technical direction and coaching engineers through the complexity yourself. Either way, you own the platform, not a slice of it.

DUTIES

• Architect end-to-end Databricks Lakehouse platforms, including medallion architecture, on AWS (Glue or Redshift).
• Design and build batch, streaming and real-time pipelines feeding AI and advanced analytics workloads.
• Own distributed systems performance: Spark tuning, query optimisation, and cost and cluster right-sizing at scale.
• Improve CI/CD, infrastructure as code (Terraform) and automated deployment practice across the platform.
• Technical Lead level: own delivery of your components directly, whether you\'re already leading others or ready to step into that for the first time.
• Principal Engineer / Hands-on Architect level: set technical direction across the build and coach engineers through the complexity, not just review their output.
• Get direct access to advanced vendor training and close working relationships with the platform partners behind the stack.

REQUIREMENTS

• Proven experience building data platforms end to end, infrastructure through to data lake, not one layer in isolation.
• Deep hands-on Databricks and Delta Lake, including medallion architecture and PySpark or Scala, with AWS (Glue or Redshift at minimum).
• Advanced SQL and performance tuning across distributed systems, with real Spark and cloud troubleshooting experience.
• Databricks certifications (Certified Data Engineer / Certified Professional) and AWS certifications highly regarded, not mandatory.
• Technical Lead level: strong individual delivery on complex builds. Prior leadership experience is valued but not required, clear potential to lead is enough.
• Principal Engineer / Hands-on Architect level: demonstrated ownership of technical direction and coaching engineers through delivery, not just managing tickets.
• Comfortable in genuine ambiguity on large, complex programs, not tightly scoped analytics or reporting projects.
• Full Australian working rights required. We are unable to progress applications without them.

KEY SKILLS

Databricks, Delta Lake, Apache Spark, PySpark, Scala, Medallion Architecture, AWS, Glue, Redshift, Terraform, CI/CD, Data Engineering, Data Architecture, Data Platform, Data Lake, Sydney.

A note from the consultant running this search: I\'ve placed engineers into this practice who came in as builders and ended up leading their own workstreams within a year. The ones who thrive aren\'t chasing a title, they\'re chasing the platform work that\'s actually hard. If your CV reads like tidy dashboards and light reporting, this isn\'t your role. If it reads like you\'ve fought with Spark cluster configs at 11pm and won, let\'s talk.

Be careful - Don\'t provide your bank or credit card details when applying for jobs. Don\'t transfer any money or complete suspicious online surveys. If you see something suspicious, report this job ad .