Azure Databricks Data Engineer: Real-Time Pipelines & Spark

Tech Mahindra; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Tech Mahindra in Sydney is hiring a hands-on Data Engineer to design, build, and optimize scalable data pipelines on Azure Databricks. The role requires strong experience with Spark, Scala, SQL, Python, and modern cloud-based data platforms.

You will implement medallion architecture with Delta Lake, develop real-time and near real-time data integrations using Spark Streaming and Kafka, and operationalize data workloads with CI/CD, Git workflows, and observability.

Qualifications

• Hands-on experience with Azure Databricks, Spark/Scala, SQL, and Python.
• Strong knowledge of data warehousing concepts and big data technologies.
• Experience with Azure and/or AWS cloud platforms.
• Familiarity with Kafka, Hive/Impala/Sqoop and related streaming tech is a plus.
• Experience with CI/CD tools like Azure DevOps, GitHub, Jenkins, Jira or Confluence.
• Excellent communication and stakeholder management skills.

Responsibilities

• Design and develop scalable ETL/ELT pipelines using Azure Databricks, Spark/Scala, and SQL.
• Build data solutions following medallion architecture and Delta Lake best practices.
• Develop real-time and near real-time data integrations using Spark Streaming and Kafka.
• Operationalize data workloads through scheduling, CI/CD, Git-based workflows, secrets management, and observability.
• Collaborate with stakeholders to deliver reliable, high-quality data engineering solutions.

Skills

Azure Databricks

Scala Spark

SQL

Python

Big data technologies

Cloud platforms (Azure/AWS)

Kafka

CI/CD tools

Communication & stakeholder mgmt

Tools

CI/CD tools: Azure DevOps, GitHub, Jenkins

Job description

Tech Mahindra in Sydney is hiring a hands-on Data Engineer to design, build, and optimize scalable data pipelines on Azure Databricks. The role requires strong experience with Spark, Scala, SQL, Python, and modern cloud-based data platforms.

You will implement medallion architecture with Delta Lake, develop real-time and near real-time data integrations using Spark Streaming and Kafka, and operationalize data workloads with CI/CD, Git workflows, and observability.