Platform Engineer: Databricks, Spark & Lakehouse

Tata Consultancy Services; New South Wales; On-site; AUD 130,000 - 190,000; Full time

Job summary

Tata Consultancy Services is seeking a skilled data engineering professional for the Australian market. The role focuses on maintaining Cloudera and Spark-based platforms, with Databricks experience and strong SQL query performance tuning in large data lakes.

You will work on Lake House architectures, Iceberg concepts, and cloud integrations using Azure Synapse. The position demands hands-on scripting in Bash/Python and infrastructure as code (Ansible/Terraform).

Qualifications

• Hands-on experience with Cloudera platform maintenance/administration.
• Experience with Databricks and Spark ecosystem.
• Deep understanding of Iceberg and Lake House architecture.
• Basic understanding of cloud environment.
• Proficiency in Bash and Python scripting.
• Experience with PySpark.
• Extensive troubleshooting of Spark code.
• Experience with Kubernetes and containerized platforms.

Skills

Cloudera

Databricks

Iceberg

Lake House

Azure Synapse

Ansible

Terraform

Bash

Python

PySpark

Spark

Kubernetes

SQL

Hive/Impala/Trino

Tools

CLI/Automation Scripting

Job description

Tata Consultancy Services is seeking a skilled data engineering professional for the Australian market. The role focuses on maintaining Cloudera and Spark-based platforms, with Databricks experience and strong SQL query performance tuning in large data lakes.

You will work on Lake House architectures, Iceberg concepts, and cloud integrations using Azure Synapse. The position demands hands-on scripting in Bash/Python and infrastructure as code (Ansible/Terraform).