Senior Databricks Data Engineer - Medallion Data Platforms

EPAM Systems Inc; Australia; On-site; AUD 150,000 - 200,000; Full time

Job summary

EPAM Systems Inc. is seeking a Senior Data Engineer to model large-scale batch and streaming data into Bronze, Silver and Gold layers using medallion architecture. You will build Databricks pipelines, develop PySpark workloads, and collaborate with stakeholders to deliver governed data products and tests.

Key focus areas include data modeling, Delta Lake governance, and AI-assisted tooling to accelerate delivery while ensuring quality and lineage.

Qualifications

• Data modeling across conceptual, logical and physical models incl. Data Vault.
• Databricks experience with Delta Lake and governance concepts like Unity Catalog.
• Streaming data experience with Spark Structured Streaming and Kafka.
• Python for pipeline development and orchestration.
• Advanced SQL for transformations and performance tuning.
• Data governance covering quality, lineage and data contracts.
• AI-assisted development tooling exposure to speed up delivery.

Responsibilities

• Design trusted Bronze, Silver and Gold layers using medallion architecture principles.
• Model large-scale batch and streaming datasets into reusable, governed data models for reporting use cases.
• Build Databricks pipelines using Delta Lake, Delta Live Tables or Lakeflow aligned to quality and lineage needs.
• Develop Spark workloads using PySpark and Spark SQL for performance, reliability and maintainability.
• Implement streaming ingestion and processing using Spark Structured Streaming and Kafka or similar.
• Partner with business and regulatory stakeholders to translate reporting needs into data products and model definitions.
• Strengthen governance with cataloging, lineage and data quality checks using Unity Catalog or equivalent patterns.
• Contribute to CI/CD for data pipelines, Git-based workflows, testing and release practices.

Skills

Data modeling

Databricks

PySpark

Spark SQL

SQL

AI-assisted tooling

Tools

Delta Lake

Delta Live Tables

Unity Catalog

Lakeflow

Apache Kafka

Job description

EPAM Systems Inc. is seeking a Senior Data Engineer to model large-scale batch and streaming data into Bronze, Silver and Gold layers using medallion architecture. You will build Databricks pipelines, develop PySpark workloads, and collaborate with stakeholders to deliver governed data products and tests.

Key focus areas include data modeling, Delta Lake governance, and AI-assisted tooling to accelerate delivery while ensuring quality and lineage.