Senior Data/ML Engineer - Forecasting Platform Modernization

Toptal; United States; Remote; USD 150,000 - 190,000; Full time

Job summary

We are seeking a Senior Data/ML Engineer at Toptal to modernize a live Databricks-based forecasting platform. You will refactor, test, and raise engineering standards for a system already delivering real business value.

Responsibilities include moving logic from notebooks to modular Python code, solidifying CI/CD, and improving monitoring/observability to ensure reliable forecasting workflows.

Qualifications

• Extensive experience with Databricks workspaces, notebooks, and scheduled workflows.
• Strong PySpark experience with production data pipelines.
• Proven ability to refactor production code for maintainability.
• Familiarity with time series forecasting concepts and workflows.
• Able to understand and work in an existing codebase.
• Experience improving testing strategies and engineering standards.
• Experience implementing automated testing practices.
• Experience improving monitoring and observability in production systems.
• Good judgment on technical debt and refactoring priorities.
• Ability to work with existing systems rather than building from scratch.

Responsibilities

• Review current forecasting platform architecture and identify improvements.
• Refactor Databricks, Python, and PySpark implementations.
• Move business logic from notebooks into reusable Python modules/packages.
• Improve separation between orchestration and core logic.
• Define engineering standards and what good looks like for the platform.
• Implement or improve automated testing for forecasting workflows.
• Build or improve monitoring and observability for predictions.
• Monitor model behavior and operational health over time.
• Improve reliability of scheduled training workflows and failure handling.
• Maintain and extend forecasting capabilities as needed.

Skills

Databricks

PySpark

Python

CI/CD

Testing strategies

Monitoring

Time series

Data pipelines

Tools

Airflow

MLflow

Job description

We are seeking a Senior Data/ML Engineer at Toptal to modernize a live Databricks-based forecasting platform. You will refactor, test, and raise engineering standards for a system already delivering real business value.

Responsibilities include moving logic from notebooks to modular Python code, solidifying CI/CD, and improving monitoring/observability to ensure reliable forecasting workflows.