Senior Databricks Data Engineer | AI-Driven Data Modeling

EPAM Systems Inc; New South Wales; On-site; AUD 180,000 - 240,000; Full time

Job summary

EPAM Systems Inc in Australia is seeking a Senior Data Engineer (Databricks, Data Modeling) to model large-scale batch and streaming data into Bronze, Silver and Gold layers for regulatory reporting data products. You will build on Databricks using Spark, Python and SQL, and partner with stakeholders on definitions while using AI-assisted tools to speed up delivery, testing and documentation.

You will design medallion architectures, develop scalable Spark pipelines, and collaborate across teams

Qualifications

• Data modeling across conceptual, logical and physical models.
• Databricks experience including Delta Lake and governance concepts.
• Spark with PySpark and Spark SQL in production.
• Streaming data with Spark Structured Streaming and Kafka.
• Python for pipeline development.
• Advanced SQL for transformations and performance.
• Data governance covering quality, lineage and data contracts.
• AI-assisted development tooling exposure (Claude or similar).

Responsibilities

• Design trusted Bronze, Silver and Gold layers using medallion architecture principles.
• Model large-scale batch and streaming datasets into reusable, governed data models for reporting use cases.
• Build Databricks pipelines using Delta Lake, Delta Live Tables or Lakeflow aligned to quality and lineage needs.
• Develop Spark workloads using PySpark and Spark SQL for performance, reliability and maintainability.
• Implement streaming ingestion and processing using Spark Structured Streaming and Kafka or similar.
• Partner with business and regulatory stakeholders to translate reporting needs into data products and model definitions.
• Strengthen governance with cataloging, lineage and data quality checks using Unity Catalog or equivalent patterns.
• Contribute to CI/CD for data pipelines, Git-based workflows, testing and release practices.

Job description

EPAM Systems Inc in Australia is seeking a Senior Data Engineer (Databricks, Data Modeling) to model large-scale batch and streaming data into Bronze, Silver and Gold layers for regulatory reporting data products. You will build on Databricks using Spark, Python and SQL, and partner with stakeholders on definitions while using AI-assisted tools to speed up delivery, testing and documentation.

You will design medallion architectures, develop scalable Spark pipelines, and collaborate across teams