Senior Data Platform Engineer - Azure & Databricks

Tata Consultancy Services; Auckland; On-site; NZD 120,000 - 180,000; Full time

Benefits offered by this job

Competitive salary

Global exposure

Structured learning and development

Career progression

Job summary

Tata Consultancy Services in Australia is seeking an experienced Data Engineer to design scalable data platform architectures on Azure and Databricks, implement medallion patterns, and craft reliable data pipelines for analytics and reporting.

You will lead legacy modernization, enforce data quality and governance, model data for analytical workloads, and collaborate with stakeholders to deliver end‑to‑end solutions while applying CI/CD practices and mentoring junior engineers.

Qualifications

• 8+ years of experience in data engineering, data platform development, or data architecture roles.
• Deep expertise in Databricks (Delta Lake, medallion architecture, Unity Catalog) and Azure Data Services (ADF, ADLS, Synapse, Azure SQL).
• Strong SQL, PySpark/Spark, and Python skills with hands-on pipeline development experience at scale.
• Proven experience designing and operating data pipelines at enterprise scale - batch and streaming.
• Solid understanding of data modelling, data governance, and data quality frameworks.
• Experience with CI/CD for data pipelines using Azure DevOps, including automated testing and release management.

Responsibilities

• Design and implement scalable data platform architectures on Azure and Databricks, including medallion patterns and lakehouse architectures.
• Build and maintain robust ETL/ELT pipelines using Databricks and Azure data services for batch and streaming workloads.
• Lead legacy data pipeline migrations to cloud-native solutions on Azure/Databricks with migration strategies and cutover plans.
• Establish data quality frameworks, lineage tracking, and access governance aligned with policies.
• Create logical and physical data models for analytic and operational workloads, optimizing for performance and cost.
• Engage with client data teams, present architectures, and support estimation and planning.
• Develop CI/CD pipelines and automated testing for data workloads; mentor engineers across teams.

Skills

Data engineering

Databricks

Azure Data Services

SQL

PySpark/Spark

Data modelling

CI/CD for data

Stakeholder communication

Tools

Azure DevOps

Job description

Tata Consultancy Services in Australia is seeking an experienced Data Engineer to design scalable data platform architectures on Azure and Databricks, implement medallion patterns, and craft reliable data pipelines for analytics and reporting.

You will lead legacy modernization, enforce data quality and governance, model data for analytical workloads, and collaborate with stakeholders to deliver end‑to‑end solutions while applying CI/CD practices and mentoring junior engineers.