Databricks Platform Administrator/Engineer – Sydney

Delivery Centric; Sydney; On-site; AUD 160,000 - 190,000; Full time

Job summary

Delivery Centric in Sydney is seeking a Senior Databricks Platform Administrator to manage enterprise Databricks environments, ensuring platform security, reliability, performance, governance, and cost optimisation. The role will supportdata, analytics, ML, and AI workloads while working closely with data, cloud, and security teams.

The role collaborates with data engineers, data scientists, MLOps, cloud, and security teams to deliver scalable compute, governance, and on-call production support.

Qualifications

• 5+ years of experience administering data or analytics platforms in enterprise environments.
• Hands-on experience administering Databricks workspaces, Spark-based platforms and clusters.
• Strong knowledge of Apache Spark architecture, execution, and performance tuning.
• Proficiency with Python and Bash scripting; SQL understanding required.
• Experience with Unity Catalog, Delta Lake, and data governance concepts.
• Cloud platform knowledge (Azure, AWS, or GCP) and cost optimisation awareness.

Responsibilities

• Administer enterprise Databricks workspaces across multiple environments.
• Configure clusters, jobs, pools, policies, secrets, and init scripts.
• Enforce cluster security, stability, and cost controls.
• Manage compute, autoscaling, and various cluster types.
• Oversee Databricks Jobs, Workflows, and scheduling for batch/streaming workloads.
• Implement and manage Unity Catalog for governance and access control.
• Integrate with cloud storage and external data sources.
• Monitor health using logs, metrics, and alerts; troubleshoot issues.
• Support ML/AI workloads with Databricks Runtime and Model Registry.
• Lead CI/CD for notebooks and configurations using Git workflows.
• Collaborate with data engineers, data scientists, MLOps, cloud and security teams.

Skills

Databricks administration

Spark architecture

Security concepts

CI/CD pipelines

Python scripting

Bash scripting

Cloud platforms

Data governance

Unity Catalog

Delta Lake

SQL

Education

Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a related discipline

Tools

Terraform

Git-based tooling

Databricks REST APIs

CLI tools

Job description

Delivery Centric is seeking a Senior Databricks Platform Administrator to manage and support enterprise Databricks environments, ensuring platform security, reliability, performance, governance, and cost optimisation. The role will supportdata, analytics, ML, and AI workloads while working closely with data, cloud, and security teams.

Key Responsibilities

• Administer and operate enterprise Databricks workspaces across development, QA, production, and disaster recovery environments.
• Manage Databricks workspace configuration including clusters, jobs, pools, policies, secrets, and init scripts.
• Design, configure, and enforce cluster policies to ensure security, stability, and cost control.
• Manage Databricks compute including autoscaling, instance pools, job clusters, and all-purpose clusters.
• Administer Databricks Jobs, Workflows, and scheduling for batch and streaming workloads.
• Configure and manage Databricks security including workspace access controls, IAM integration, permissions, and secrets management.
• Implement and manage Databricks Unity Catalog for data governance, access control, and lineage.
• Configure and support integration with cloud storage (Azure Data Lake, S3, GCS) and external data sources.
• Administer Delta Lake features including table management, versioning, optimization, and retention policies.
• Monitor platform health, performance, and availability using logs, metrics, and alerts.
• Troubleshoot Spark workloads including job failures, performance degradation, memory issues, and scaling problems.
• Support ML and AI workloads using Databricks ML Runtime, Model Registry, Feature Store, and notebooks.
• Implement CI/CD processes for Databricks notebooks, jobs, and configurations using Git-based workflows.
• Manage Databricks REST APIs, CLI tools, and automation scripts for platform operations.
• Perform capacity planning, usage analysis, and cost optimisation across Databricks environments.
• Apply platform patches, runtime upgrades, and configuration changes following change management best practices.
• Act as the escalation point for Databricks production incidents and lead root cause analysis.
• Maintain operational documentation, standards, and platform runbooks.
• Collaborate with data engineers, data scientists, MLOps, cloud, and security teams.
• Provide on-call production support through a rotation schedule.

Qualifications & Experience

• 5+ years of experience administering data or analytics platforms in enterprise environments.
• Strong hands-on experience administering Databricks workspaces and Spark-based platforms.
• Deep understanding of Apache Spark architecture, execution, and performance tuning.
• Experience managing Databricks clusters, jobs, workflows, and autoscaling configurations.
• Strong understanding of Databricks security concepts including workspace access controls, IAM integration, and secrets management.
• Hands-on experience with Unity Catalog and data governance concepts.
• Experience working with Delta Lake and lakehouse architectures.
• Strong knowledge of cloud platforms (Azure, AWS, or GCP) and cloud storage services.
• Proficiency in Python and Bash scripting.
• Good understanding of SQL and distributed data processing concepts.
• Experience implementing CI/CD pipelines for data platforms using Git-based tooling.
• Strong troubleshooting, analytical, and problem-solving skills.
• Strong communication and stakeholder engagement skills.
• Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a related discipline.

Preferred Skills

• Databricks Administrator or Data Engineer certification.
• Experience supporting ML and AI workloads on Databricks.
• Experience with streaming technologies such as Spark Structured Streaming or Kafka.
• Familiarity with Infrastructure as Code tools such as Terraform for Databricks.
• Experience working in regulated or security-sensitive environments.
• Familiarity with ITIL-based enterprise service management processes.

Delivery Centric is seeking a Senior Databricks Platform Administrator to manage and support enterprise Databricks environments, ensuring platform security, reliability, performance, governance, and cost optimisation. The role will supportdata, analytics, ML, and AI workloads while working closely with data, cloud, and security teams.

Key Responsibilities

• Administer and operate enterprise Databricks workspaces across development, QA, production, and disaster recovery environments.
• Manage Databricks workspace configuration including clusters, jobs, pools, policies, secrets, and init scripts.
• Design, configure, and enforce cluster policies to ensure security, stability, and cost control.
• Manage Databricks compute including autoscaling, instance pools, job clusters, and all-purpose clusters.
• Administer Databricks Jobs, Workflows, and scheduling for batch and streaming workloads.
• Configure and manage Databricks security including workspace access controls, IAM integration, permissions, and secrets management.
• Implement and manage Databricks Unity Catalog for data governance, access control, and lineage.
• Configure and support integration with cloud storage (Azure Data Lake, S3, GCS) and external data sources.
• Administer Delta Lake features including table management, versioning, optimization, and retention policies.
• Monitor platform health, performance, and availability using logs, metrics, and alerts.
• Troubleshoot Spark workloads including job failures, performance degradation, memory issues, and scaling problems.
• Support ML and AI workloads using Databricks ML Runtime, Model Registry, Feature Store, and notebooks.
• Implement CI/CD processes for Databricks notebooks, jobs, and configurations using Git-based workflows.
• Manage Databricks REST APIs, CLI tools, and automation scripts for platform operations.
• Perform capacity planning, usage analysis, and cost optimisation across Databricks environments.
• Apply platform patches, runtime upgrades, and configuration changes following change management best practices.
• Act as the escalation point for Databricks production incidents and lead root cause analysis.
• Maintain operational documentation, standards, and platform runbooks.
• Collaborate with data engineers, data scientists, MLOps, cloud, and security teams.
• Provide on-call production support through a rotation schedule.

Qualifications & Experience

• 5+ years of experience administering data or analytics platforms in enterprise environments.
• Strong hands-on experience administering Databricks workspaces and Spark-based platforms.
• Deep understanding of Apache Spark architecture, execution, and performance tuning.
• Experience managing Databricks clusters, jobs, workflows, and autoscaling configurations.
• Strong understanding of Databricks security concepts including workspace access controls, IAM integration, and secrets management.
• Hands-on experience with Unity Catalog and data governance concepts.
• Experience working with Delta Lake and lakehouse architectures.
• Strong knowledge of cloud platforms (Azure, AWS, or GCP) and cloud storage services.
• Proficiency in Python and Bash scripting.
• Good understanding of SQL and distributed data processing concepts.
• Experience implementing CI/CD pipelines for data platforms using Git-based tooling.
• Strong troubleshooting, analytical, and problem-solving skills.
• Strong communication and stakeholder engagement skills.
• Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a related discipline.

Preferred Skills

• Databricks Administrator or Data Engineer certification.
• Experience supporting ML and AI workloads on Databricks.
• Experience with streaming technologies such as Spark Structured Streaming or Kafka.
• Familiarity with Infrastructure as Code tools such as Terraform for Databricks.
• Experience working in regulated or security-sensitive environments.
• Familiarity with ITIL-based enterprise service management processes.