Senior Data/ML Engineer — Databricks Forecasting Platform | Remote

Neura Market; Toronto; Hybrid; CAD 120,000 - 165,000; Full time

Job summary

Neura Market is seeking a Senior Data/ML Engineer to refactor and operationalize an existing Databricks-based forecasting platform already delivering business value. You’ll reduce technical debt, strengthen tests, and raise engineering standards for a live system.

The role emphasizes migrating logic from notebooks to reusable Python modules and improving observability. You will work on production-grade time series forecasting pipelines, with a focus on maintainability, reliability, and scalable

Qualifications

• Experienced with Databricks environments including workspaces, notebooks, scheduled workflows, and CI/CD
• Strong Python engineering with design of reusable modules or packages
• Hands-on PySpark with production data pipelines or distributed processing
• Experience refactoring production code to improve maintainability and extensibility
• Familiarity with time series forecasting workflows and concepts
• Ability to understand and work effectively within an unfamiliar codebase
• Experience improving testing strategy and software quality in production systems
• Experience implementing automated testing practices
• Experience improving monitoring/observability for production systems
• Ability to prioritize technical debt and define maintainable architecture

Responsibilities

• Review the current forecasting platform architecture and identify improvement areas
• Refactor Databricks, Python, and PySpark implementations
• Move business logic from notebooks into reusable Python modules
• Improve separation of orchestration and core business logic
• Set engineering standards and define what good looks like for the platform
• Implement or improve automated testing and validation for forecasting workflows
• Build or improve monitoring and observability of predictions
• Monitor model behavior and operational health over time
• Improve reliability of scheduled training workflows
• Enhance failure handling, retries, and workflow resilience
• Maintain and extend forecasting capabilities as needed

Skills

Databricks

Python

PySpark

CI/CD

Refactoring

Time series forecasting

Testing practices

Monitoring/observability

Production code maintainability

Understanding legacy codebase

Tools

Databricks notebooks

Python modules/packages

CI/CD pipelines

Job description

Headquarters: Remote
URL: https://www.toptal.com/

About the Role

We're looking for a Senior Data/ML Engineer to refactor, operationalize, and improve an existing time series forecasting platform that's already live and driving real business value. This is not a greenfield build — the focus is modernizing a Databricks-based forecasting system that's been maintained primarily by a single developer for years. You'll reduce technical debt, strengthen testing and observability, and raise the engineering bar on a system the business already depends on. If you'd rather bring discipline and maturity to an existing production system than start from a blank slate, this is built for that.

What You'll Do

• Review the current forecasting platform architecture and identify areas for improvement

• Refactor existing Databricks, Python, and PySpark implementations

• Move business logic out of Databricks notebooks and into reusable Python modules or packages

• Improve separation of concerns between orchestration and core business logic

• Establish stronger engineering standards and help define what "good" looks like for the platform

• Implement or improve automated testing practices and validation mechanisms for forecasting workflows

• Build or improve monitoring and observability, increasing visibility into how predictions are generated

• Help monitor model behavior and operational health over time

• Improve reliability of scheduled training workflows, reducing manual intervention on failure

• Improve failure handling, retries, and overall workflow resilience

• Maintain and extend existing forecasting capabilities as needed

What You Bring

• Strong professional experience with Databricks, including workspaces, notebooks, scheduled workflows, and CI/CD processes

• Strong Python engineering experience, including designing reusable modules or packages

• Strong PySpark experience with production data pipelines or distributed data processing

• Experience refactoring production code and improving maintainability

• Familiarity with time series forecasting concepts and workflows

• Ability to understand and work effectively within an existing, unfamiliar codebase

• Experience improving software quality, testing strategy, and engineering standards

• Experience implementing automated testing practices

• Experience improving monitoring, observability, or operational visibility for production systems

• Strong judgment around technical debt, refactoring priorities, and maintainable architecture

• Ability to work with existing systems rather than only building from scratch

Why This Role

• Real production impact: Improve a system the business already relies on, not a proof-of-concept

• Engineering maturity focus: Bring testing, observability, and maintainability to a platform that’s outgrown its current state

• Meaningful ownership: Help define engineering standards for the forecasting platform going forward

• Flexible location: Preference for Toronto or St. Louis, but open to remote consultants globally with North American working-hours overlap