Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health; New York (NY); On-site; USD 180,000 - 240,000; Full time

Job summary

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design, build, and maintain systems that measure and improve the quality of AI-powered applications. You’ll establish engineering infrastructure, evaluation methods, and monitoring practices enabling safe, reliable AI features across patient experiences and merchant tools.

This hands-on role sits at the intersection of ML engineering and software reliability, requiring collaboration with applied AI engineers

Qualifications

• 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline.
• Experience developing and operating machine learning or AI systems in production.
• Strong programming skills in Python and experience building production software services.
• Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.
• Strong understanding of machine learning evaluation methodologies and performance measurement.
• Experience developing reproducible experiments and benchmark datasets.

Responsibilities

• Design and build evaluation infrastructure for AI and machine learning applications across Bask Health.
• Develop automated testing systems for large language model applications and intelligent workflows.
• Establish measurable quality standards for production AI features.
• Create evaluation pipelines that assess accuracy, relevance, and reliability.
• Build and maintain representative datasets for testing AI application behavior.
• Develop regression testing systems to detect quality changes when models or prompts are updated.
• Design evaluation methods for retrieval-augmented generation and grounded AI systems.
• Measure retrieval quality, grounding accuracy, and unsupported model outputs.

Skills

Python

Production services

Automated testing

LLM architectures

Evaluation pipelines

Monitoring observability

Tools

Cursor

Claude Code

LangSmith

Job description

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design, build, and maintain systems that measure and improve the quality of AI-powered applications. You’ll establish engineering infrastructure, evaluation methods, and monitoring practices enabling safe, reliable AI features across patient experiences and merchant tools.

This hands-on role sits at the intersection of ML engineering and software reliability, requiring collaboration with applied AI engineers