Senior Data Engineer - Databricks

ClearCompany; City of Melbourne; On-site; AUD 100,000 - 130,000; Full time

Job summary

A leading retail business is seeking a Senior Data Engineer to support the development of an Azure Databricks platform. The successful candidate will design and develop data pipelines, optimise workflows, and manage CI/CD workflows. Extensive experience in Azure Databricks, PySpark, and data pipelines is essential. This position is based in Victoria, Australia.

Qualifications

• Extensive experience building data pipelines in an Azure Databricks environment.
• Well-versed with PySpark and SQL.
• Solid experience around ML models and algorithms.

Responsibilities

• Design and develop pipelines using Databricks and Lakehouse.
• Optimise and maintain data workflows, ensuring quality and integrity.
• Performance tuning and monitoring.
• Set up and manage CI/CD pipelines.
• Notebook development using Python and/or Pyspark.
• Adhere to best practices for data engineering, including governance and security.
• Own ML engines, further developing architecture for model execution and monitoring.

Skills

Azure Databricks

PySpark

SQL

Data Pipeline Development

Machine Learning Models

Job description

Senior Data Engineer required to support the development of an Azure Databricks platform for a leading retail business.

Responsibilities:

• design and development of pipelines using Databricks and Lakehouse
• optimisation and maintenance of data workflows, ensuring quality and integrity
• performance tuning and monitoring
• set up and manage CI/CD pipelines
• Notebook development using Python and/or Pyspark
• adhere best practices for data engineering, including governance and security
• own ML engines, further developing architecture for model execution and monitoring

Requirements:

• extensive experience building data pipelines in an Azure Databricks environment
• well-versed with PySpark and SQL
• solid experience around ML models and algorithms

Click on the 'Apply' button to submit your CV.