Senior Databricks & AI Architect

Recmatrix Consulting; Sydney; On-site; AUD 180,000 - 260,000; Full time

Job summary

Recmatrix Consulting seeks a Senior Databricks & AI Architect to lead design and implementation of data and AI solutions using Databricks.

The role focuses on data engineering, platform optimisation, FinOps and emerging AI capabilities like Agentic AI, LLMs and RAG, with an emphasis on Insurance or Financial Services sectors.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Science, IT, Engineering or related field.
• Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional preferred.
• 8+ years of experience in Data Engineering, Data Architecture, or Solution Architecture.

Responsibilities

• Design end-to-end data architectures using the Databricks platform.
• Lead scalable Spark, Delta Lake, and Databricks SQL solutions.
• Develop Medallion Architecture and reusable ETL/ELT frameworks.

Skills

Databricks

Spark

Delta Lake

AI/ML

LangChain

LLMs

RAG

FinOps

Data governance

Education

Bachelor’s or Master’s degree in Computer Science, Data Science, IT, Engineering

Tools

Databricks

Delta Lake

Unity Catalog

Photon

LlamaIndex

Job description

Job Role: Senior Databricks & AI Architect

Location: Sydney, Australia

Employment Type: Contract

Experience: 8+ Years

Role Overview

We are looking for a Senior Databricks & AI Architect to lead the design and implementation of modern data and AI solutions. The role will focus on Databricks architecture, data engineering, platform optimisation, FinOps, and emerging AI capabilities including Agentic AI, LLMs, and RAG.

The ideal candidate will have strong hands-on Databricks experience, excellent knowledge of Spark and Delta Lake, and experience delivering data and AI solutions within Insurance or Financial Services.

Key Responsibilities

• Design and govern end-to-end data architectures using the Databricks Data Intelligence Platform.
• Lead the design of scalable Spark, Delta Lake, and Databricks SQL solutions.
• Develop and standardise Medallion Architecture and reusable ETL/ELT frameworks.
• Design data models using Star Schema, Snowflake Schema, and Data Vault 2.0.
• Architect AI solutions using Generative AI, Agentic AI, LLMs, and RAG patterns.
• Design multi-agent workflows and LLM orchestration using LangChain, LlamaIndex, or similar frameworks.
• Develop AI solutions for insurance use cases such as claims, underwriting, policy management, and risk assessment.
• Drive Databricks performance optimisation and FinOps, including cluster policies, instance selection, Photon, Z-Ordering, and Liquid Clustering.
• Implement cost monitoring, attribution, and optimisation strategies across Databricks workloads.
• Ensure data governance, security, access control, and lineage using Databricks Unity Catalog.
• Work closely with business, engineering, data science, security, and architecture teams to deliver scalable solutions.
• Provide technical leadership, architectural guidance, and mentoring to data engineering and AI teams.

Required Skills & Experience

• Bachelor’s or Master’s degree in Computer Science, Data Science, Information Technology, Engineering, or a related discipline.
• Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional is preferred.
• 8+ years of experience in Data Engineering, Data Architecture, or Solution Architecture.
• 3+ years of hands-on Databricks experience, including technical/architectural leadership.
• Strong expertise in Apache Spark, PySpark and/or Scala.
• Strong hands-on experience with Databricks, Delta Lake, and Databricks SQL.
• Experience with Medallion Architecture and enterprise data modelling.
• Strong knowledge of Star Schema, Snowflake Schema, and/or Data Vault 2.0.
• Proven experience delivering AI/ML and Generative AI solutions.
• Strong understanding of Agentic AI, LLM orchestration, and RAG.
• Experience with LangChain and/or LlamaIndex.
• Strong knowledge of Databricks FinOps, performance tuning, and cost optimisation.
• Hands-on experience with Photon, Z-Ordering, Liquid Clustering, cluster policies, and workload optimisation.
• Strong understanding of Unity Catalog, data governance, security, and access controls.
• Experience working with structured and unstructured data.
• Strong communication and stakeholder management skills.

Domain Experience

• Experience in Insurance or Financial Services is highly preferred.
• Understanding of insurance data and processes such as policies, premiums, claims, underwriting, actuarial models, and risk assessment is desirable.
• Experience working in regulated enterprise environments is an advantage.