Senior ML Engineer, AI Evaluation & Reliability

Bask Health

United States

Remote

USD 150,000 - 210,000

Full time

6 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design, build, and maintain systems that measure and improve the quality of AI-powered applications. You’ll establish evaluation infrastructure, pipelines, and monitoring practices to ship AI features with confidence, across patient experiences, merchant tools, and clinical workflows.

This hands-on senior role combines ML engineering, software infrastructure, automated testing, and production reliability, with

Qualifications

  • 5+ years of professional experience in machine learning engineering or a closely related technical discipline.
  • Demonstrated experience operating ML/AI systems in production.
  • Strong programming skills in Python and building production software services.
  • Experience designing automated testing frameworks or evaluation pipelines for AI/ML apps.
  • Familiarity with large language models and modern LLM architectures.
  • Understanding of prompt engineering and model inference patterns.
  • Experience with retrieval-augmented generation and evaluation methods.
  • Experience with data pipelines and analytical workflows.
  • Understanding of software testing, regression testing, and CI/CD.

Responsibilities

  • Design and build evaluation infrastructure for AI/ML applications across Bask Health.
  • Develop automated testing systems for large language model apps and intelligent workflows.
  • Establish measurable quality standards for production AI features.
  • Create evaluation pipelines assessing accuracy, relevance, completeness, consistency, and reliability.
  • Build and maintain representative datasets for testing AI behavior.
  • Develop regression testing systems to detect quality changes when models or prompts update.
  • Design evaluation methods for retrieval-augmented generation and knowledge-grounded AI systems.
  • Measure retrieval quality, grounding accuracy, and unsupported outputs.
  • Build tools to identify hallucinations and unreliable responses.
  • Develop testing approaches for AI agents, tool calling, and multi-step workflows.
  • Evaluate whether AI systems follow instructions and respect workflow boundaries.
  • Implement automated checks for structured outputs, API interactions, and tool execution.
  • Build monitoring systems tracking AI performance in production.
  • Develop dashboards for model quality, latency, errors, and reliability.
  • Investigate production AI failures and identify root causes.
  • Collaborate to improve prompts, pipelines, and model configurations.
  • Establish testing procedures for model upgrades and provider changes.
  • Develop benchmarking tools comparing models on quality and cost.
  • Implement evaluation processes for sensitive healthcare AI workflows.
  • Work with product and clinical stakeholders to define acceptance criteria and escalation needs.
  • Support human review and feedback collection for outputs needing validation.
  • Design reproducible experiments for evaluating AI improvements.
  • Build reusable evaluation libraries and internal tools.
  • Integrate AI evaluation checks into development and deployment workflows.
  • Improve observability through logging, tracing, and diagnostics.
  • Mentor engineers on AI testing methodologies and reliability practices.
  • Contribute to architecture discussions for AI infrastructure, testing, and monitoring.
  • Own evaluation and reliability improvements from design to deployment.

Skills

Python programming
ML engineering
Evaluation pipelines
Model monitoring
Software testing
CI/CD
Debugging production systems
Communication
Experiment design

Tools

LangSmith
DeepEval
Ragas

Job description

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design, build, and maintain systems that measure and improve the quality of AI-powered applications. You’ll establish evaluation infrastructure, pipelines, and monitoring practices to ship AI features with confidence, across patient experiences, merchant tools, and clinical workflows.

This hands-on senior role combines ML engineering, software infrastructure, automated testing, and production reliability, with

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior ML Reliability Engineer - AI Evaluation & HealthTech
Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health • New York (NY)

On-site
USD 180,000 - 240,000
AI Evaluation & Reliability Engineer
AI Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 140,000 - 200,000
Remote Senior LLM Evaluation & Reliability Engineer
Remote Senior LLM Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 160,000 - 220,000
AI Reliability Engineer — Evaluation & Testing
AI Reliability Engineer — Evaluation & Testing

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
AI Evaluation Platform Engineer
AI Evaluation Platform Engineer

Bask Health • New York (NY)

On-site
USD 140,000 - 190,000
AI Evaluation & Quality Assurance Engineer
AI Evaluation & Quality Assurance Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
AI Quality & Evaluation Engineer
AI Quality & Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
Healthcare AI Quality Assurance Engineer
Healthcare AI Quality Assurance Engineer

Bask Health • United States

Remote
USD 120,000 - 180,000
AI Quality & Evaluation Engineer — Backend Tools
AI Quality & Evaluation Engineer — Backend Tools

Bask Health • United States

Remote
USD 120,000 - 180,000
AI-First ML Systems Engineer
AI-First ML Systems Engineer

Bask Health • United States

Remote
USD 120,000 - 160,000