Senior Databricks Data Engineer – Medallion Architecture

EPAM Systems Inc; Sydney; On-site; AUD 150,000 - 190,000; Full time

Job summary

EPAM Systems Inc seeks a Senior Data Engineer with Databricks and data modeling expertise to design and implement scalable data platforms. You will build trusted Bronze/Silver/Gold layers, develop Spark workloads, and ensure governance across data products in collaboration with business stakeholders.

Responsibilities include implementing streaming ingestion, building pipelines with Delta Lake, and contributing to CI/CD processes. Experience with AI-assisted tooling is a plus.

Qualifications

• Strong data modeling across conceptual, logical and physical models.
• Databricks experience with Delta Lake and governance concepts such as Unity Catalog.
• Spark/PySpark production experience.
• Streaming experience with Structured Streaming and Kafka or similar.
• Advanced SQL for transformations, validation and performance tuning.
• Exposure to AI-assisted development tooling preferred to accelerate build and testing.

Responsibilities

• Design Bronze, Silver and Gold medallion layers for governance.
• Model large-scale batch and streaming datasets for reporting data products.
• Build Databricks pipelines aligned to quality and lineage needs (Delta Lake / Delta Live Tables).
• Develop Spark workloads using PySpark and Spark SQL for performance and maintainability.
• Implement streaming ingestion with Spark Structured Streaming and Kafka.
• Partner with stakeholders to translate reporting needs into data products and definitions.
• Strengthen governance with cataloging, lineage and data quality checks (Unity Catalog or equivalents).
• Contribute to CI/CD for data pipelines, Git-based workflows, testing and release practices.

Skills

Databricks

Delta Lake

PySpark

Spark SQL

Data modeling

Data governance

Kafka

Python

Tools

Delta Live Tables

Lakeflow

Unity Catalog

Job description

EPAM Systems Inc seeks a Senior Data Engineer with Databricks and data modeling expertise to design and implement scalable data platforms. You will build trusted Bronze/Silver/Gold layers, develop Spark workloads, and ensure governance across data products in collaboration with business stakeholders.

Responsibilities include implementing streaming ingestion, building pipelines with Delta Lake, and contributing to CI/CD processes. Experience with AI-assisted tooling is a plus.