Databricks Architect

Paxus; City of Melbourne; On-site; AUD 170,000 - 210,000; Full time

Job summary

Paxus is seeking an experienced Databricks Architect to design, implement, and optimise scalable enterprise data platforms and AI/ML architectures using the Databricks platform.

You will lead architecture across Azure, AWS or GCP, shape lakehouse patterns, implement CI/CD and IaC, and collaborate with data engineers, security teams and stakeholders to drive enterprise adoption.

Qualifications

• Extensive experience designing enterprise data platforms and lakehouse architectures.
• Strong hands-on Databricks architecture and implementation experience.
• Advanced knowledge of Apache Spark, PySpark, Spark SQL and performance optimisation.
• Strong experience with Delta Lake and Unity Catalog.
• Experience designing ETL/ELT pipelines and distributed data-processing solutions.
• Strong cloud architecture experience across Azure, AWS or GCP.
• Experience with cloud storage, networking, identity and security services.
• Knowledge of data governance, lineage, access controls and regulatory compliance.
• Experience implementing CI/CD and infrastructure-as-code using Terraform, Azure DevOps, GitHub Actions or Jenkins.
• Strong understanding of SQL, Python and data-modelling principles.
• Experience integrating Databricks with enterprise data and analytics platforms.
• Strong stakeholder-management and technical-leadership capabilities.

Responsibilities

• Lead end-to-end architecture and deployment of large-scale data platforms on Databricks.
• Design scalable batch and real-time data processing and ETL/ELT pipelines.
• Develop medallion and lakehouse patterns for enterprise workloads.
• Architect secure, resilient Databricks environments across cloud providers.
• Establish data governance, lineage, and Unity Catalog standards.
• Define integrations between Databricks and storage, databases, APIs, BI tools.
• Implement CI/CD and infrastructure-as-code for Databricks workloads.
• Optimize Spark workloads, clusters and SQL performance for cost efficiency.
• Support ML/GenAI solutions and operationalisation.
• Provide technical leadership and architecture governance.

Skills

Databricks

Lakehouse

Spark SQL

PySpark

CI/CD

Terraform

Unity Catalog

Data governance

MLOps

RBAC

Cloud platforms

Tools

Azure DevOps

GitHub Actions

Jenkins

Databricks

Job description

We are seeking an experienced Databricks Architect to design, implement and optimise scalable enterprise data platforms, distributed processing solutions and AI/ML architectures using the Databricks platform.

Key Responsibilities

• Lead the end-to-end architecture and deployment of large-scale data platforms using Databricks, Apache Spark, Delta Lake and Unity Catalog.
• Design scalable batch and real-time data processing solutions and ETL/ELT pipelines.
• Develop medallion and lakehouse architecture patterns for enterprise data workloads.
• Architect secure and resilient Databricks environments across Azure, AWS or GCP.
• Establish data governance, access controls, lineage and compliance standards through Unity Catalog.
• Design integrations between Databricks and cloud storage, databases, APIs, BI platforms and enterprise applications.
• Define platform security standards covering identity, networking, encryption, secrets management and role-based access.
• Implement CI/CD, infrastructure-as-code and automated deployment practices for Databricks workloads.
• Optimise Spark jobs, clusters, SQL workloads and resource utilisation for performance and cost efficiency.
• Support the architecture and operationalisation of machine learning and generative AI solutions.
• Troubleshoot complex platform, integration and performance issues.
• Create reusable architecture patterns, standards and technical documentation.
• Collaborate with data engineers, AI/ML teams, security specialists, cloud engineers and business stakeholders.
• Provide technical leadership, architecture assurance and mentoring to engineering teams.
• Develop platform roadmaps and drive enterprise adoption of Databricks.

Mandatory Skills and Experience

• Extensive experience designing enterprise data platforms and lakehouse architectures.
• Strong hands-on Databricks architecture and implementation experience.
• Advanced knowledge of Apache Spark, PySpark, Spark SQL and performance optimisation.
• Strong experience with Delta Lake and Unity Catalog.
• Experience designing ETL/ELT pipelines and distributed data-processing solutions.
• Strong cloud architecture experience across Azure, AWS or GCP.
• Experience with cloud storage, networking, identity and security services.
• Knowledge of data governance, lineage, access controls and regulatory compliance.
• Experience implementing CI/CD and infrastructure-as-code using tools such as Terraform, Azure DevOps, GitHub Actions or Jenkins.
• Strong understanding of SQL, Python and data-modelling principles.
• Experience integrating Databricks with enterprise data and analytics platforms.
• Strong stakeholder-management and technical-leadership capabilities.

Highly Desirable

• Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
• Databricks Certified Generative AI Engineer or related architecture certification.
• Experience with MLflow, Feature Store, Model Serving and MLOps.
• Experience designing AI, machine-learning, GenAI or agentic AI solutions.
• Knowledge of streaming technologies such as Kafka, Structured Streaming or Event Hubs.
• Experience delivering regulated, government or large-enterprise data platforms.

Desired Skills and Experience

• Databricks Architect
• Led the architecture, design, and implementation of enterprise-scale Databricks Lakehouse platforms leveraging Apache Spark, Delta Lake, and Unity Catalog.
• Designed and delivered scalable batch and real-time data processing solutions, enabling high-performance analytics and business intelligence capabilities.
• Developed enterprise data architectures using Medallion and Lakehouse design patterns to support advanced data engineering, analytics, and AI workloads.
• Architected secure, resilient, and scalable Databricks environments across Azure, AWS, and GCP cloud platforms.
• Established enterprise data governance frameworks, implementing access controls, lineage, compliance standards, and data security through Unity Catalog.
• Designed integrations between Databricks, cloud storage services, databases, APIs, enterprise applications, and BI platforms.
• Defined platform security standards covering identity management, networking, encryption, secrets management, and role-based access controls (RBAC).
• Implemented CI/CD pipelines and Infrastructure-as-Code solutions using Terraform, Azure DevOps, GitHub Actions, and Jenkins to automate deployments and platform management.
• Optimized Apache Spark workloads, cluster configurations, and SQL performance to improve processing efficiency and reduce operational costs.
• Supported the architecture and operationalization of Machine Learning, MLOps, and Generative AI solutions using MLflow, Feature Store, and Model Serving capabilities.
• Provided technical leadership, architectural governance, mentoring, and best-practice guidance to cross-functional engineering teams.
• Collaborated with data engineers, AI/ML specialists, cloud engineers, security teams, and business stakeholders to deliver strategic data initiatives.
• Troubleshot and resolved complex platform, integration, and performance challenges across enterprise environments.
• Developed reusable architecture standards, reference models, and technical documentation to accelerate delivery and support platform adoption.
• Created strategic platform roadmaps and drove enterprise-wide adoption of Databricks technologies and modern data architectures.

Paxus values diversity and welcomes applications from Indigenous Australians, people from diverse cultural and linguistic backgrounds and people living with a disability. If you require an adjustment to the recruitment process, including the application form in an alternate format, please contact me on the above contact details.