Data Engineer

Blackbook.AI; City of Brisbane; Hybrid; AUD 100,000 - 160,000; Full time

Benefits offered by this job

Competitive Remuneration

Flexible Working Conditions

Gym and PT

Free Spotify Account

Job summary

Blackbook.ai is a leading Australian AI company delivering data-driven outcomes for clients. We are seeking a Data Engineer to design, build and operate modern cloud data platforms across Microsoft Fabric and Databricks, with pipelines, warehousing and reporting functions.

You will work directly with clients, gather requirements and implement scalable data transformations in Python, SQL and PySpark while applying best practices and collaborating with cross-functional teams.

Qualifications

• Bachelor's degree or higher.
• Proficiency in Python and SQL for data engineering.
• Hands-on experience with Microsoft Fabric (Lakehouse, Warehouse, Notebooks and Pipelines).
• Building and orchestrating pipelines in Azure Data Factory or Fabric Pipelines.
• Cloud data warehousing using medallion architecture (bronze/silver/gold).
• Designing and building star schema dimensional models.
• Commercial experience with Databricks and PySpark.
• Reporting and data visualization, ideally with Power BI.
• Appropriate use of AI tools in data engineering work.
• Strong presentation and communication skills in writing and speaking.
• Critical thinking and problem solving abilities.
• Experience with AWS data engineering or willingness to upskill.

Responsibilities

• Work directly with clients as a Blackbook.ai consultant.
• Gather requirements, design, build and test data and analytics solutions.
• Build and orchestrate data pipelines using Fabric Pipelines and Azure Data Factory.
• Develop scalable data transformations in Python, PySpark and SQL across Fabric and Databricks.
• Design and implement cloud data warehouses and lakehouses with medallion architecture.
• Build star schema dimensional models for scalable, cost-efficient reporting.
• Deliver reporting and semantic models that business users can trust.
• Design and build metadata-driven data ingestion into data lakes, lakehouses and databases.
• Use AI tools to accelerate delivery and extract value from AI.
• Apply engineering best practices: Git, code reviews, automated testing, CI/CD.
• Take ownership of end-to-end process from requirements to implementation.
• Keep up with emerging technologies and share knowledge with peers.

Skills

Python

SQL

Problem solving

Communication

Education

Bachelor's degree in a related field

Tools

Microsoft Fabric

Azure Data Factory

Databricks

PySpark

Power BI

Git

CI/CD

Job description

Blackbook.ai is one of Australia's most successful and progressive AI companies. Our use of Data and AI to deliver on our client's pursuit of bottom-line benefits has resulted in a company that can claim a 100% success rate in solution delivery. We have an agile approach to tackling problems. We understand that there isn't just one solution to every problem! Our unique approach to problem-solving will see you working directly with our customers, capturing additional requirements and developing intuitive solutions while drawing on the vast knowledge of existing team members.

The Data Engineer role will see you working with multiple clients, projects and stakeholders alongside fellow experts in the DataOps, Digital and AI teams. You will be hands-on building modern cloud data platforms, primarily on Microsoft Fabric and Databricks, covering pipeline engineering, data modelling and reporting, with the opportunity to expand into AWS as our client base grows.

Your responsibilities as a Data Engineer are as follows:

Work directly with clients as a Blackbook.ai consultant

Gather requirements, design, build and test data and analytics solutions

Build and orchestrate data pipelines using Microsoft Fabric Pipelines and Azure Data Factory

Develop scalable data transformations in Python, PySpark and SQL across Fabric and Databricks

Design and implement cloud data warehouses and lakehouses using medallion (bronze, silver, gold) architecture

Build star schema dimensional models that support scalable, cost-efficient reporting and analytics

Deliver reporting and semantic models that business users can trust

Design and build metadata-driven data ingestion into data lakes, lakehouses and databases

Use AI tools appropriately to accelerate delivery and help clients get value from AI

Apply engineering best practice: Git, code review, automated testing and CI/CD

Take ownership of the end-to-end process from requirements gathering to implementation

Keep up with emerging technologies and share knowledge with your peers

To be considered for the position you must have a bachelor's degree or higher, and be comfortable demonstrating your skills in the following areas:

Python and SQL for data engineering (mandatory)

Hands-on experience with Microsoft Fabric (Lakehouse, Warehouse, Notebooks and Pipelines)

Building and orchestrating pipelines in Azure Data Factory or Fabric Pipelines

Data warehousing in the cloud using medallion architecture

Designing and building star schema dimensional models

Commercial experience with Databricks and PySpark / Apache Spark

Reporting and data visualisation, ideally with Power BI

Using AI tools and techniques appropriately in data engineering work

Professional presentation and communication skills: written and verbal, formal documentation, diagramming and modelling

Critical thinking and problem solving skills

Highly regarded although not mandatory experience includes:

AWS data engineering (e.g. S3, Glue, Redshift, Lambda), or a willingness to upskill as our AWS client base grows

R for data transformation and analysis

Microsoft Fabric, Azure or Databricks certification in Data Engineering

Delta Lake, Unity Catalog or MLflow

Spark performance tuning: partitioning, joins, caching and cluster sizing

Terraform, PowerShell or Azure/AWS CLI for managing cloud infrastructure

In return, we can offer you:

Competitive Remuneration

Flexible Working Conditions

Gym and PT

Free Spotify Account

Blackbook.ai can offer a fast-paced and collaborative delivery environment with an emphasis on culture and teamwork, ensuring all projects are delivered with our high standards. We have a large team of dedicated consultants able to work alongside and guide you through your journey, and offer on-the-job training and development with access to multiple certifications.