AI Evaluation Platform Engineer

Bask Health; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Bask Health seeks an ML Platform Engineer, Evaluation Systems to build and maintain the infrastructure that allows Bask Health to evaluate machine learning models and AI-powered applications reliably, consistently, and at scale. You’ll help develop platform services, data pipelines, execution environments, and monitoring capabilities that support ML evaluation across engineering teams.

This is a hands-on role focused on operational infrastructure behind evaluation systems.

Qualifications

• 3+ years of professional experience in software engineering, machine learning engineering, platform engineering, or a closely related technical field.
• Experience building and maintaining production backend services or infrastructure.
• Strong programming skills in Python, TypeScript, or comparable languages.
• Familiarity with machine learning development and evaluation workflows.
• Experience developing APIs and backend service integrations.
• Understanding of distributed systems and asynchronous processing.

Responsibilities

• Design, build, and maintain machine learning evaluation infrastructure across Bask Health.
• Develop platform services that execute and manage automated model evaluation workloads.
• Build scalable systems for running large batches of model and AI application evaluations.
• Implement job orchestration, scheduling, and execution management for evaluation pipelines.
• Develop infrastructure for storing and versioning evaluation datasets.
• Build systems for tracking model configurations, prompts, parameters, and evaluation results.
• Implement reproducible experiment workflows that support consistent model comparisons.
• Develop automated pipelines for benchmarking machine learning models.
• Build APIs and services that allow engineering teams to submit and monitor evaluation jobs.
• Create infrastructure for comparing evaluation results across model versions and experiments.
• Support evaluation workflows for large language models, predictive models, and retrieval systems.
• Develop asynchronous processing systems for computationally intensive evaluation tasks.
• Implement monitoring for evaluation job performance, failures, latency, and resource usage.
• Build logging and tracing capabilities that help engineers investigate evaluation problems.
• Develop systems for managing evaluation artifacts, reports, and historical results.
• Integrate evaluation infrastructure with model registries and ML development workflows.

Skills

Python

TypeScript

Distributed systems

Machine learning

Backend services

Cloud infrastructure

CI/CD

Testing

Git

Education

Bachelor's degree in CS or related field

Tools

Docker

Kubernetes

Airflow

MLflow

Weights & Biases

AWS

Job description

Bask Health seeks an ML Platform Engineer, Evaluation Systems to build and maintain the infrastructure that allows Bask Health to evaluate machine learning models and AI-powered applications reliably, consistently, and at scale. You’ll help develop platform services, data pipelines, execution environments, and monitoring capabilities that support ML evaluation across engineering teams.

This is a hands-on role focused on operational infrastructure behind evaluation systems.