Senior Data Engineer: Azure Databricks & ML Pipelines

ClearCompany; City of Melbourne; On-site; AUD 100,000 - 120,000; Full time

Job summary

A leading retail business is seeking a Senior Data Engineer to support the development of an Azure Databricks platform. The role involves designing and developing data pipelines, optimising workflows, and ensuring data quality and integrity. Ideal candidates should have extensive experience in Azure Databricks, proficiency in PySpark and SQL, and knowledge of ML models. This position offers an opportunity to own ML engines and work with best practices in data engineering.

Qualifications

• Extensive experience building data pipelines in Azure Databricks.
• Well-versed with PySpark and SQL.
• Solid experience around ML models and algorithms.

Responsibilities

• Design and develop pipelines using Databricks and Lakehouse.
• Optimise and maintain data workflows, ensuring quality and integrity.
• Performance tuning and monitoring.
• Set up and manage CI/CD pipelines.
• Notebook development using Python and/or Pyspark.
• Adhere to best practices for data engineering, including governance and security.
• Own ML engines, further developing architecture for model execution and monitoring.

Skills

Azure Databricks

PySpark

SQL

ML models and algorithms

Job description

A leading retail business is seeking a Senior Data Engineer to support the development of an Azure Databricks platform. The role involves designing and developing data pipelines, optimising workflows, and ensuring data quality and integrity. Ideal candidates should have extensive experience in Azure Databricks, proficiency in PySpark and SQL, and knowledge of ML models. This position offers an opportunity to own ML engines and work with best practices in data engineering.