Senior Data Engineer (Databricks, Data Modeling)

EPAM Systems Inc

Sydney

On-site

AUD 140,000 - 190,000

Full time

32 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Great Place To Work-Certified
Glassdoor Top 100 Best Places to Work
Most Loved Workplace (Newsweek 2021-23
Voice-driven ideas culture

Job summary

EPAM Systems Inc. seeks a Senior Data Engineer to model large-scale data across Bronze, Silver and Gold layers using medallion architecture.

You will build Databricks pipelines with Delta Lake and Delta Live Tables, develop Spark workloads in PySpark and Spark SQL, and implement streaming with Structured Streaming and Kafka. You’ll collaborate with stakeholders to deliver governed data products and strengthen data quality and lineage.

Qualifications

  • Data modeling background across conceptual, logical and physical models using dimensional, Data Vault or similar approaches.
  • Databricks experience, including Delta Lake plus governance concepts such as Unity Catalog.
  • Apache Spark skill sets using PySpark and Spark SQL in production environments.
  • Streaming data experience using Spark Structured Streaming, Kafka or similar technologies.
  • Python capability for pipeline development and orchestration patterns.
  • Advanced SQL skill sets for transformation logic, validation and performance tuning.
  • Data governance experience covering quality, lineage, controls and well-defined data contracts.
  • AI-assisted development tooling exposure, with Claude preferred, to accelerate build, testing and documentation.

Responsibilities

  • Design trusted Bronze, Silver and Gold layers using medallion architecture principles.
  • Model large-scale batch and streaming datasets into reusable, governed data models for reporting use cases.
  • Build Databricks pipelines using Delta Lake, Delta Live Tables or Lakeflow aligned to quality and lineage needs.
  • Develop Spark workloads using PySpark and Spark SQL for performance, reliability and maintainability.
  • Implement streaming ingestion and processing using Spark Structured Streaming and Kafka or similar.
  • Partner with business and regulatory stakeholders to translate reporting needs into data products and model definitions.
  • Strengthen governance with cataloging, lineage and data quality checks using Unity Catalog or equivalent patterns.
  • Contribute to CI/CD for data pipelines, Git-based workflows, testing and release practices.

Skills

Data modeling
Databricks
Spark (PySpark)
Streaming
Python
SQL
Data governance
AI-assisted tooling

Tools

Delta Lake
Delta Live Tables
Lakeflow
Unity Catalog
Kafka

Job description

We are seeking a Senior Data Engineer (Databricks, Data Modeling). You will model large-scale batch and streaming data into governed Bronze, Silver and Gold layers to support regulatory reporting data products. You will build on Databricks using Spark, Python and SQL, partner with stakeholders on definitions and use AI-assisted tools to speed up delivery, testing and documentation.

Responsibilities
  • Design trusted Bronze, Silver and Gold layers using medallion architecture principles
  • Model large-scale batch and streaming datasets into reusable, governed data models for reporting use cases
  • Build Databricks pipelines using Delta Lake, Delta Live Tables or Lakeflow aligned to quality and lineage needs
  • Develop Spark workloads using PySpark and Spark SQL for performance, reliability and maintainability
  • Implement streaming ingestion and processing using Spark Structured Streaming and Kafka or similar
  • Partner with business and regulatory stakeholders to translate reporting needs into data products and model definitions
  • Strengthen governance with cataloging, lineage and data quality checks using Unity Catalog or equivalent patterns
  • Contribute to CI/CD for data pipelines, Git-based workflows, testing and release practices
Requirements
  • Data modeling background across conceptual, logical and physical models using dimensional, Data Vault or similar approaches
  • Databricks experience, including Delta Lake plus governance concepts such as Unity Catalog
  • Apache Spark skill sets using PySpark and Spark SQL in production environments
  • Streaming data experience using Spark Structured Streaming, Kafka or similar technologies
  • Python capability for pipeline development and orchestration patterns
  • Advanced SQL skill sets for transformation logic, validation and performance tuning
  • Data governance experience covering quality, lineage, controls and well-defined data contracts
  • AI-assisted development tooling exposure, with Claude preferred, to accelerate build, testing and documentation
Nice to have
  • Regulatory reporting data exposure such as EMIR, MiFIR or ASIC derivative reporting
  • Data mesh experience delivering domain-owned data products
  • Data Vault 2.0 experience in large-scale enterprise models
  • Apache Kafka experience and strong understanding of streaming platform operations
We offer
  • By choosing EPAM, you're getting a job at Great Place To Work-Certified™ in 2024, Glassdoor’s Top 100 Best Places to Work in 2023 & one of Most Loved Workplace, as recognized by Newsweek, 2021 - 2023.
  • Employee ideas are the main driver of our business. We have a very supportive environment where your voice matters.
  • You will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practices.
  • We offer a transparent career path and an individual roadmap to engineer your future & accelerate your journey.
  • At EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with us.
Life at EPAM
  • We are a leading global provider of digital platform engineering and development services that has rapidly grown in Australia since 2013.
  • We adopt a global growth strategy, thinking and acting like start-ups, working in multi-disciplinary teams, and delivering results. With a multinational talented team, we have been delivering innovative solutions for cloud migration, cyber security, digital transformation, etc.
  • You will work in an open and friendly work environment where your professional skills and experience can stand out.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer (Databricks, Data Modeling)
Senior Data Engineer (Databricks, Data Modeling)

EPAM Systems • Sydney

On-site
AUD 150,000 - 190,000
Data And Ai Architect
Data And Ai Architect

EPAM Systems Inc • Melbourne

On-site
AUD 180,000 - 280,000
Great Place To Work 2024
Top 100 Best Places to Work 2023
Mentoring & Certification support
Data and AI Architect
Data and AI Architect

EPAM Systems Inc • Melbourne

On-site
AUD 140,000 - 210,000
Great Place To Work 2024 recognition
Global talent collaboration
Online courses and mentoring programs
Databricks Data & AI Architect for IoT & RAG
Databricks Data & AI Architect for IoT & RAG

EPAM Systems Inc • Melbourne

On-site
AUD 140,000 - 210,000
Great Place To Work 2024 recognition
Global talent collaboration
Online courses and mentoring programs
Senior Manager, Data Analytics and AI Consulting
Senior Manager, Data Analytics and AI Consulting

EPAM Systems Inc • Sydney

On-site
AUD 180,000 - 240,000
Great Place To Work-Certified 2024
Mentoring programs
Access to online courses & libraries
Lead Data & AI Engineer
Lead Data & AI Engineer

EPAM Systems • Australia

On-site
AUD 150,000 - 190,000
Senior Python Engineer (AI-Assisted SDLC)
Senior Python Engineer (AI-Assisted SDLC)

EPAM Systems Inc • Sydney

On-site
AUD 140,000 - 180,000
Senior Data Engineer (Databricks, Data Modeling)
Senior Data Engineer (Databricks, Data Modeling)

EPAM Systems • Australia

On-site
AUD 150,000 - 200,000
Senior Databricks Data Engineer – Medallion Architecture
Senior Databricks Data Engineer – Medallion Architecture

EPAM Systems Inc • Sydney

On-site
AUD 150,000 - 190,000
Senior Data Engineer — Databricks, Medallion Modeling
Senior Data Engineer — Databricks, Medallion Modeling

EPAM Systems Inc • Sydney

On-site
AUD 140,000 - 190,000
Great Place To Work-Certified
Glassdoor Top 100 Best Places to Work
Most Loved Workplace (Newsweek 2021-23
+1
Top locations