Databricks Tech Lead | Azure Data Architect | Sydney

Tata Consultancy Services; Sydney; On-site; AUD 180,000 - 240,000; Full time

Job summary

Tata Consultancy Services Australia is seeking a Databricks Engineer (Tech Lead) in Sydney to lead the design, development and delivery of large-scale data engineering solutions on Databricks.

You will architect and implement data pipelines using PySpark, Delta Lake, Unity Catalog, and Databricks Workflows, mentor teams, and collaborate with stakeholders to translate requirements into scalable, secure platforms.

Qualifications

• Strong hands-on experience with Databricks, PySpark, Spark SQL, and Delta Lake.
• Experience with Azure Databricks and related Azure services (ADF, ADLS, Synapse, Key Vault, Entra ID).
• Expertise in data modelling, ETL/ELT design, and large-scale distributed data processing.
• Strong SQL and Python development skills.
• Experience with CI/CD pipelines, Git, Azure DevOps, and Infrastructure as Code.
• Experience implementing data governance and security controls.

Responsibilities

• Lead the design, development, and delivery of large-scale data engineering solutions on Databricks.
• Architect and implement data pipelines using PySpark, Delta Lake, Unity Catalog, DLT, and Databricks Workflows.
• Drive migration and modernization initiatives from legacy platforms to Databricks.
• Collaborate with business stakeholders, architects, analysts, and data consumers to translate requirements into scalable solutions.
• Provide technical leadership, code reviews, design guidance, and mentoring to engineering teams.
• Establish best practices for CI/CD, data quality, monitoring, performance optimization, and security.
• Support production releases, troubleshooting, and platform governance.

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Azure Databricks

ADF

ADLS

Synapse

Key Vault

Entra ID

SQL

Python

CI/CD

Git

Azure DevOps

Infrastructure as Code

Data governance

Security controls

Tools

Unity Catalog

Delta Live Tables

DLT

Databricks Asset Bundles

Job description

Tata Consultancy Services Australia is seeking a Databricks Engineer (Tech Lead) in Sydney to lead the design, development and delivery of large-scale data engineering solutions on Databricks.

You will architect and implement data pipelines using PySpark, Delta Lake, Unity Catalog, and Databricks Workflows, mentor teams, and collaborate with stakeholders to translate requirements into scalable, secure platforms.