Senior Data & Analytics Engineer - Databricks

Balance Recruitment and Balance Consulting; New South Wales; On-site; AUD 180,000 - 230,000; Full time

Benefits offered by this job

Great Place to Work

Greenfield project

Career development

Job summary

Balance Recruitment and Balance Consulting seeks a Senior Data & Analytics Engineer to design and implement robust data pipelines in Databricks, with a focus on data modelling for multiple business workstreams in Western Sydney. You will lead end-to-end data engineering, optimise performance of Spark jobs, and build scalable solutions for sales, operations, marketing and finance reporting.

A hands-on role requiring strong Python (PySpark) and SQL skills and ability to deliver trusted data for

Qualifications

• Experience designing end-to-end data pipelines in Databricks.
• Strong data modelling skills including star/snowflake schemas.
• Proficient in Python and SQL for data engineering tasks.

Responsibilities

• Design, build and maintain scalable data pipelines in Databricks (PySpark/SQL).
• Develop robust data models unifying ERP and other data sources.
• Implement BI layers to enable accurate, single-source reporting.

Skills

Data Modelling

Python (PySpark)

SQL

Tools

Databricks

BI Tool

Job description

Seeking a Senior Data & Analytics Engineer with recent Databricks for this rapidly expanding and successful business in Western Sydney!

Looking for a high-performing hands on Data Engineer to design and implement robust data pipelines using Databricks with a focus on Data Modelling for sales, operations marketing and financial workstreams.

Key responsibilities

• Delivering solutions: Defining the problem statements, conducting the business analysis and building requirements, design and delivering solutions
• Pipeline Architecture: Design, build, and maintain scalable end-to-end data pipelines in Databricks (PySpark/SQL). Includes orchestrate Medallion-architecture pipelines in Databricks (Unity Catalog, Delta Live Tables).
• Advanced Data Modelling: Develop star/snowflake schemas and vault models that unify disparate data from various ERP sources.
• Domain Expertise: Build complex logic for sales, operations and financial reporting, including and analytics (pipeline health, churn, conversion).
• BI Delivery: Develop and optimise high-performance semantic layers in to ensure "one version of the truth."
• Training users: provide training to users
• Performance Tuning: Optimise large-scale Spark jobs to ensure lightning-fast reporting for stakeholders. Includes optimisation, indexing strategies, and partition management to reduce latency in BI reports.
• Security & Governance: Implement Row-Level Security (RLS) and Object-Level Security (OLS) across Databricks and SQL environments to ensure sensitive financial data is restricted.

Success metrics

• Business value: User satisfaction and utilisation of the data services by our users in running the day-to-day operations
• Data Reliability: Zero "Data Mismatch" incidents reported by the Finance team during month-end closing.
• Operational Excellence: Implementation of automated alerts for pipeline failures and database performance bottlenecks
• Cost Efficiency: Reduction in monthly Databricks/Cloud spend through optimised SQL and cluster management.
• Reporting Speed: load times for the primary finance & Sales dashboards.

Required skills

• Core Tech: Solid Data Engineering experience with a focus on Databricks (Medallion architecture)
• Data Modelling: Proven expertise in dimensional modelling and translating complex business processes into logical data structures.
• BI Tools: Expert-level proficiency in a major BI tool for complex calculated fields.
• Programming: Strong proficiency in Python (PySpark) and SQL.

What's in it for you?

• A friendly, professional environment that has been recognised as a 'Great Place to Work' for five years consecutively
• A greenfield opportunity to make your mark on a true Business & Technology Transformation
• Outstanding career development and learning
• Work on multiple CRM and ERP platforms including SAP S/4HANA Public Cloud