Senior ML Engineer: Scalable Pipelines & MLOps

Bailey Abbott Pty Ltd; Perth; On-site; AUD 120,000 - 180,000; Full time

Job summary

Bailey Abbott Pty Ltd in Perth, Australia is seeking a Machine Learning Engineer with a strong software engineering foundation to productionise, scale, and maintain advanced analytical and predictive models.

You will bridge the gap between data science experimentation and robust software engineering, designing scalable data pipelines and automated ML lifecycles within our Databricks environment using Spark, R, and modern engineering practices.

Qualifications

• Solid foundation in core software engineering principles (design patterns, unit/integration testing, Git version control, CI/CD pipelines).
• Proven hands-on experience orchestrating workloads in Databricks (MLflow, Unity Catalog).
• Deep practical understanding of Spark internals, data partitioning, query optimization, and memory management.
• Strong proficiency in R for production analytics using Spark integrations (sparklyr, SparkR).
• Experience querying relational data and working with Delta Lake formats.

Responsibilities

• Production ML pipelines: Build, test, and optimize production-grade data processing and model training pipelines in Apache Spark and Databricks.
• MLOps & lifecycle automation: Standardise model deployment, versioning, monitoring, and automated retraining workflows (CI/CD, MLflow).
• Code quality & architecture: Apply software engineering best practices—modular code, automated testing, containerisation, and version control.
• R & statistical workloads: Optimise and scale R-based modeling workflows over distributed data using Sparklyr/SparkR.
• Collaboration & integration: Partner with data architects and engineers to integrate predictive outputs into downstream analytics.
• Performance & reliability: Monitor pipeline latency, manage compute sizing, ensure data governance in production.

Skills

Software engineering fundamentals

Databricks ecosystem

Distributed computing (Apache Spark)

R for production / advanced analytics

Data layer experience

Tools

MLflow

Unity Catalog

Sparklyr / SparkR

Delta Lake

Job description

Bailey Abbott Pty Ltd in Perth, Australia is seeking a Machine Learning Engineer with a strong software engineering foundation to productionise, scale, and maintain advanced analytical and predictive models.

You will bridge the gap between data science experimentation and robust software engineering, designing scalable data pipelines and automated ML lifecycles within our Databricks environment using Spark, R, and modern engineering practices.