Senior Databricks / Machine Learning Engineer

Supreme Consulting; Hyderabad, Bengaluru; On-site; INR 7,842,000 - 13,070,000; Part time

Job summary

Supreme Consulting seeks an experienced Senior Databricks / Machine Learning Engineer for a remote, part-time assignment. You will contribute 3 hours daily, within the 18:00–10:00 IST support window.

The role requires strong hands-on Azure, Azure Databricks, Apache Spark, PySpark and ML proficiency, with data ingestion, processing and deployment workflow experience. You will implement ML models with Python/Keras, use MLflow for experiments and registry, manage Databricks Jobs, and collaborate

Qualifications

• 8+ years IT experience overall.
• Strong Azure Databricks experience.
• Strong Machine Learning experience.
• Strong Python + PySpark + Apache Spark skills.
• Azure Data Lake Storage experience.
• Azure Synapse Pipelines experience.
• Keras ML model development experience.
• Optuna hyperparameter tuning experience.
• MLflow + MLflow Model Registry experience.
• Databricks Jobs + YAML experience.
• GitHub experience.
• Delta / Parquet experience.
• Understanding of ML lifecycle and deployment.
• Experience with production/operational systems.
• Available for 3 hours daily.
• Available within 6 PM-10 AM support window.

Responsibilities

• Develop and support Azure Databricks data and ML solutions.
• Build scalable data processing pipelines using Spark/PySpark.
• Work with Azure Synapse Pipelines for data ingestion.
• Process and manage data in ADLS using Delta/Parquet formats.
• Develop ML solutions using Python and Keras.
• Implement hyperparameter optimization using Optuna.
• Manage ML experiments and model lifecycle using MLflow.
• Maintain MLflow Model Registry.
• Configure Databricks Jobs and YAML-based workflows.
• Use GitHub for source control and JFrog Artifactory for artifacts.
• Support deployment of models to plant servers.
• Work with plant telemetry, equipment metadata, DTR and BOPH systems.
• Support operational analysis with Genie agents and future custom-agent ecosystems.
• Troubleshoot data, ML and deployment issues.
• Maintain technical documentation in Confluence.

Skills

Azure Databricks

Python

PySpark

Apache Spark

Machine Learning

Keras

Optuna

MLflow

GitHub

Delta Lake/Parquet

Databricks Jobs

YAML

Tools

Azure Databricks

Apache Spark

PySpark

GitHub

MLflow

MLflow Model Registry

Databricks Jobs

Delta Lake

Parquet

Job description

• Support Window: 6:00 PM-10:00 AM
• Assignment: 3 Hours Daily
• Mode: Remote / Part-Time
• Focus: Databricks + Machine Learning

ROLE OVERVIEW

• We are looking for an experienced Senior Databricks / Machine Learning Engineer with 8+ years of experience for a part-time job support assignment.
• The ideal candidate must have strong hands‑on experience in Azure, Azure Databricks, Apache Spark, PySpark and Machine Learning, with experience building data ingestion, processing, ML experimentation and deployment workflows.
• Candidates must be available for a 3-hour assignment within the 6:00 PM
• 10:00 AM support window.
• MUST-HAVE TECHNOLOGIES - Cloud: Microsoft Azure Data Ingestion: Azure Synapse Pipelines Storage: Azure Data Lake Storage (ADLS) Data Processing: Azure Databricks Apache Spark PySpark Data Formats: Delta Lake / Parquet Machine Learning: Python Keras Machine Learning model development Hyperparameter Tuning: Optuna Experiment Tracking: MLflow Model Registry: MLflow Model Registry Workflow Orchestration: Databricks Jobs YAML Configurations Source Control: GitHub Artifact Management: JFrog Artifactory Documentation: Confluence Operational / AI Analysis: Genie Agents Future custom-agent ecosystem Deployment: Plant Servers Supporting Systems: DTR BOPH / Equipment Metadata Plant Telemetry Systems

KEY RESPONSIBILITIES

• Develop and support Azure Databricks data and ML solutions.
• Build scalable data processing pipelines using Spark/PySpark.
• Work with Azure Synapse Pipelines for data ingestion.
• Process and manage data in ADLS using Delta/Parquet formats.
• Develop ML solutions using Python and Keras.
• Implement hyperparameter optimization using Optuna.
• Manage ML experiments and model lifecycle using MLflow.
• Maintain MLflow Model Registry.
• Configure and manage Databricks Jobs and YAML-based workflows.
• Use GitHub for source control and JFrog Artifactory for artifact management.
• Support deployment of models/solutions to plant servers.
• Work with plant telemetry, equipment metadata, DTR and BOPH systems.
• Support operational analysis involving Genie agents and future custom-agent ecosystems.
• Troubleshoot data, ML and deployment issues.
• Maintain technical documentation in Confluence.

CANDIDATE MUST HAVE

• 8+ years overall IT experience
• Strong Azure Databricks experience
• Strong Machine Learning experience
• Strong Python + PySpark + Apache Spark
• Azure Data Lake Storage experience
• Azure Synapse Pipelines experience
• Keras / ML model development experience
• Optuna / hyperparameter tuning experience
• MLflow + MLflow Model Registry experience
• Databricks Jobs + YAML experience
• GitHub experience
• Experience with Delta / Parquet
• Good understanding of ML lifecycle and deployment
• Experience working with production/operational systems
• Available for 3 hours daily
• Available within 6 PM-10 AM support window.