Machine Learning Engineer: Production ML & Data Pipelines

Bailey Abbott Pty Ltd; Western Australia; On-site; AUD 120,000 - 180,000; Full time

Job summary

Bailey Abbott Pty Ltd in Australia is seeking a Machine Learning Engineer with a strong software engineering foundation to productionise, scale, and maintain advanced analytical models. You will bridge data science experimentation with robust software practices, designing scalable data pipelines and automated ML lifecycles within our Databricks environment using Spark and R.

Collaborate with data architects and engineers to integrate predictive outputs into downstream systems, monitor pipeline

Qualifications

• Solid foundation in core software engineering principles (design patterns, testing, CI/CD).
• Hands-on Databricks experience with MLflow/Unity Catalog.
• Deep understanding of Spark internals, partitioning, optimization, memory management.
• Strong proficiency in R for production analytics and integration with Spark (sparklyr/SparkR).
• Experience with large-scale time-series or tabular data on distributed architectures.

Responsibilities

• Build, test, and optimize production-grade ML pipelines in Spark and Databricks.
• Standardise model deployment, versioning, monitoring, and retraining workflows (CI/CD, MLflow).
• Apply best practices for modular code, automated tests, containerisation, and version control.
• Scale R-based modeling workflows over distributed data architectures using Sparklyr/SparkR.
• Collaborate with data architects and engineers to integrate outputs into downstream systems.
• Monitor pipeline latency, manage compute cluster sizing, and ensure data governance across spaces.

Skills

Software engineering fundamentals

Databricks ecosystem

Distributed computing (Apache Spark)

R for production / advanced analytics

Data layer experience

Tools

MLflow

Unity Catalog

Delta Lake

Sparklyr

Job description

Bailey Abbott Pty Ltd in Australia is seeking a Machine Learning Engineer with a strong software engineering foundation to productionise, scale, and maintain advanced analytical models. You will bridge data science experimentation with robust software practices, designing scalable data pipelines and automated ML lifecycles within our Databricks environment using Spark and R.

Collaborate with data architects and engineers to integrate predictive outputs into downstream systems, monitor pipeline