Azure - Databricks

BULLIT MANAGEMENT SERVICES LIMITED; Sydney; On-site; AUD 140,000 - 190,000; Full time

Job summary

BULLIT MANAGEMENT SERVICES LIMITED in Sydney is looking for a Senior Data Engineer to design and optimise large-scale data platforms and data pipelines. You will work with data science teams to deliver actionable insights and ensure data quality across the organisation.

The role requires deep experience with PySpark, Python, Azure Synapse, ADF, Azure Functions and data warehousing concepts, plus familiarity with distributed systems and DevOps basics across the development lifecycle.

Qualifications

• 8+ years in development/software engineering with Agile experience.
• Strong Python and SQL skills; Java is a plus.
• Hands-on with complex data pipelines and distributed systems.
• Extensive use of PySpark, Azure Synapse, ADF, Azure Functions; data models and ETL pipelines.

Skills

Software development

Agile experience

Python

SQL

Java experience

Distributed systems

Data pipelines

MLOps

Data modelling

Data warehousing

DevOps basics

ETL methods

Tools

PySpark

Azure Synapse

Azure Data Factory (ADF)

Azure Functions

Azure DevOps

Git

Informatica

MS-SQL

ADLS Gen2

Big Data

Job description

• Minimum 8+ years in development/software engineering and has extensively knowledge of working in Agile organisations.
• Strong Python and SQL skill and good to have Java experience.
• Have hands-on experience building complex data pipelines,
• Have extensive experience with Apache PySpark, Python, Azure Synapse, ADF, Azure Functions, data models and ETL pipelines.
• Have experience in setting up and optimizing both SQL and NoSQL data stores (MS-SQL, ADLS Gen2), as well as familiarity with object storage services (e.g., ADLS Gen2, Big Data),
• Experience in managing and further developing distributed systems and clusters for batch processing,
• Knowledge of MLOps architecture and practices,
• Experience with modern data modelling concepts and large-scale data warehousing architecture.
• Basic understanding of modern DevOps tool set, e.g. Azure DevOps, Git.
• Knowledge of ETL and data warehousing methods, good to have Informatica experience.