Senior Data Engineer (Databricks, Data Modeling)

EPAM Systems Inc; Sydney; On-site; AUD 150,000 - 190,000; Full time

Job summary

EPAM Systems Inc seeks a Senior Data Engineer with Databricks and data modeling expertise to design and implement scalable data platforms. You will build trusted Bronze/Silver/Gold layers, develop Spark workloads, and ensure governance across data products in collaboration with business stakeholders.

Responsibilities include implementing streaming ingestion, building pipelines with Delta Lake, and contributing to CI/CD processes. Experience with AI-assisted tooling is a plus.

Qualifications

• Strong data modeling across conceptual, logical and physical models.
• Databricks experience with Delta Lake and governance concepts such as Unity Catalog.
• Spark/PySpark production experience.
• Streaming experience with Structured Streaming and Kafka or similar.
• Advanced SQL for transformations, validation and performance tuning.
• Exposure to AI-assisted development tooling preferred to accelerate build and testing.

Responsibilities

• Design Bronze, Silver and Gold medallion layers for governance.
• Model large-scale batch and streaming datasets for reporting data products.
• Build Databricks pipelines aligned to quality and lineage needs (Delta Lake / Delta Live Tables).
• Develop Spark workloads using PySpark and Spark SQL for performance and maintainability.
• Implement streaming ingestion with Spark Structured Streaming and Kafka.
• Partner with stakeholders to translate reporting needs into data products and definitions.
• Strengthen governance with cataloging, lineage and data quality checks (Unity Catalog or equivalents).
• Contribute to CI/CD for data pipelines, Git-based workflows, testing and release practices.

Skills

Databricks

Delta Lake

PySpark

Spark SQL

Data modeling

Data governance

Kafka

Python

Tools

Delta Live Tables

Lakeflow

Unity Catalog

Job description

We are seeking a Senior Data Engineer (Databricks, Data Modeling). You will model large-scale batch and streaming data into governed Bronze, Silver and Gold layers to support regulatory reporting data products. You will build on Databricks using Spark, Python and SQL, partner with stakeholders on definitions and use AI-assisted tools to speed up delivery, testing and documentation.ResponsibilitiesDesign trusted Bronze, Silver and Gold layers using medallion architecture principlesModel large-scale batch and streaming datasets into reusable, governed data models for reporting use casesBuild Databricks pipelines using Delta Lake, Delta Live Tables or Lakeflow aligned to quality and lineage needsDevelop Spark workloads using PySpark and Spark SQL for performance, reliability and maintainabilityImplement streaming ingestion and processing using Spark Structured Streaming and Kafka or similarPartner with business and regulatory stakeholders to translate reporting needs into data products and model definitionsStrengthen governance with cataloging, lineage and data quality checks using Unity Catalog or equivalent patternsContribute to CI/CD for data pipelines, Git-based workflows, testing and release practicesRequirementsData modeling background across conceptual, logical and physical models using dimensional, Data Vault or similar approachesDatabricks experience, including Delta Lake plus governance concepts such as Unity CatalogApache Spark skill sets using PySpark and Spark SQL in production environmentsStreaming data experience using Spark Structured Streaming, Kafka or similar technologiesPython capability for pipeline development and orchestration patternsAdvanced SQL skill sets for transformation logic, validation and performance tuningData governance experience covering quality, lineage, controls and well-defined data contractsAI-assisted development tooling exposure, with Claude preferred, to accelerate build, testing and documentationNice to haveRegulatory reporting data exposure such as EMIR, MiFIR or ASIC derivative reportingData mesh experience delivering domain-owned data productsData Vault 2.0 experience in large-scale enterprise modelsApache Kafka experience and strong understanding of streaming platform operationsWe offerBy choosing EPAM, you're getting a job at Great Place To Work-Certified in 2024, Glassdoor’s Top 100 Best Places to Work in 2023 & one of Most Loved Workplace, as recognized by Newsweek, 2021 - 2023.Employee ideas are the main driver of our business. We have a very supportive environment where your voice matters.You will be challenged while working side-by-side with the best talent globally. We work with top-notch technologies, constantly seeking new industry trends and best practices.We offer a transparent career path and an individual roadmap to engineer your future & accelerate your journey.At EPAM, you can find vast opportunities for self-development: online courses and libraries, mentoring programs, partial grants of certification, and experience exchange with colleagues around the world. You will learn, contribute, and grow with us.Life at EPAM- We are a leading global provider of digital platform engineering and development services that has rapidly grown in Australia since 2013.- We adopt a global growth strategy, thinking and acting like start-ups, working in multi-disciplinary teams, and delivering results. With a multinational talented team, we have been delivering innovative solutions for cloud migration, cyber security, digital transformation, etc.- You will work in an open and friendly work environment where your professional skills and experience can stand out.How We Hire• Here, we summarize the typical journey to finding a job within EPAM.• Apply and tell us about yourself!• Go through some standard interviews:- General interview with a recruiter- Technical interview with our technology experts- Manager interview or Offer interview with a hiring manager• Get ready to join the team!• Not sure if you meet all the requirements? No problem. Let's talk anyway and find out more!• It takes 1 min of application to start the journey with us. Apply now!