Sr. Data Engineer with ML

Accord Technologies Inc; New York (NY); Hybrid; USD 120,000 - 150,000; Full time

Job summary

A technology solutions provider in New York is looking for a Sr. Data Engineer with experience in MLflow and Databricks to manage the machine learning lifecycle. The ideal candidate will design hybrid infrastructures, enable experiment tracking, and collaborate with data scientists. This long-term W2 contract position offers an opportunity to work with both on-premises and cloud environments, ensuring best practices in scalable and reliable model deployment.

Qualifications

• Proven experience with MLflow, Databricks, and open-source machine learning frameworks.
• Strong understanding of data engineering and cloud computing.
• Experience with deploying machine learning models in hybrid environments.

Responsibilities

• Design, implement, and maintain a hybrid ML infrastructure using Databricks.
• Enable experiment tracking and lifecycle management using MLflow.
• Manage registration and deployment of machine learning models.

Skills

MLflow

Databricks

Data engineering

Model lifecycle management

Python

Apache Spark

Linux/Unix systems

Collaboration

Communication

Job description

Title: Sr. Data Engineer

Location: NYC, NY

Duration: long term

Position type: W2 contract

Job Description

Data Engineer with Python and hands on experience using MLFlow - the project is to set-up experiments using MLFlow with ability to save artifacts, use MLFlow API's, configure MLFlow and Databrics functioning for running experiments..

We are seeking a skilled Hybrid Data Engineer to manage and optimize our machine learning lifecycle across a hybrid environment of on-premises open-source tools and cloud-based platforms.

The ideal candidate will have experience working with MLflow, Databricks, and open-source machine learning frameworks, enabling seamless integration, experiment tracking, model management, and deployment.

Responsibilities

• Design, implement, and maintain a hybrid ML infrastructure that connects on-premises open-source tools with cloud platforms like Databricks.
• Enable experiment tracking, model versioning, and lifecycle management using MLflow integrated with Databricks notebooks.
• Manage the registration, versioning, and deployment of machine learning models across environments.
• Collaborate with data scientists and ML engineers to streamline workflows and ensure scalable, reliable model deployment.
• Configure MLflow servers and APIs to connect securely and efficiently with existing data sources and deployment targets.
• Optimize data pipelines and model deployment processes for both on-premises and cloud environments.
• Ensure best practices for reproducibility, security, and scalability in ML workflows.

Qualifications

• Proven experience with MLflow, Databricks, and open-source machine learning frameworks.
• Strong understanding of data engineering, model lifecycle management, and cloud computing.
• Experience with deploying and managing machine learning models in hybrid environments.
• Knowledge of Apache Spark, Python, and Linux/Unix systems.
• Excellent collaboration and communication skills.