Azure Data Engineer

BULLIT RECRUITMENT SOLUTIONS LIMITED; Sydney; Vor Ort; AUD 140.000 - 180.000; Vollzeit

Zusammenfassung

BULLIT RECRUITMENT SOLUTIONS LIMITED in Sydney, Australia seeks a senior software data engineer with 8+ years in development and Agile environments to design and optimize large data pipelines. You will leverage Python and SQL, with Java experience a plus, to deliver scalable data platforms.

You will work with PySpark, Azure Synapse, ADF, and Azure Functions, building robust data models and ETL pipelines, while managing distributed batch processing systems and applying modern data warehousing

Qualifikationen

• 8+ years in development/software engineering and experience in Agile organisations.
• Strong Python and SQL skills; Java experience is a plus.
• Hands-on in building complex data pipelines and distributed systems.
• Experience with PySpark, Azure Synapse, ADF, Azure Functions, data models and ETL pipelines.

Aufgaben

• Develop and optimize complex data pipelines.
• Work with PySpark, Azure Synapse, ADF, Azure Functions and data models for ETL pipelines.
• Set up and optimize SQL and NoSQL data stores (MS-SQL, ADLS Gen2).
• Manage and evolve distributed systems and batch processing clusters.
• Apply MLOps architecture and practices.
• Implement modern data modeling and large-scale data warehousing.

Kenntnisse

Python

SQL

Agile

Java experience

Tools

Apache PySpark

Azure Synapse

Azure Data Factory (ADF)

Azure Functions

MS-SQL

ADLS Gen2

Jobbeschreibung

• Minimum 8+ years in development/software engineering and has extensively knowledge of working in Agile organisations.
• Strong Python and SQL skill and good to have Java experience.
• Have hands-on experience building complex data pipelines,
• Have extensive experience with Apache PySpark, Python, Azure Synapse, ADF, Azure Functions, data models and ETL pipelines.
• Have experience in setting up and optimizing both SQL and NoSQL data stores (MS-SQL, ADLS Gen2), as well as familiarity with object storage services (e.g., ADLS Gen2, Big Data),
• Experience in managing and further developing distributed systems and clusters for batch processing,
• Knowledge of MLOps architecture and practices,
• Experience with modern data modelling concepts and large-scale data warehousing architecture.
• Basic understanding of modern DevOps tool set, e.g. Azure DevOps, Git.
• Knowledge of ETL and data warehousing methods, good to have Informatica experience.