Remote Senior LLM Evaluation & Reliability Engineer

Bask Health

United States

Remote

USD 160,000 - 220,000

Full time

6 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design and maintain systems that measure and improve AI quality across healthcare workflows.

You'll build evaluation pipelines, regression tests, and monitoring to ensure safe, reliable AI features. This remote US role offers ownership of critical reliability practices and collaboration with product and clinical stakeholders.

Qualifications

  • 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline.
  • Demonstrated experience developing and operating machine learning or AI systems in production.
  • Strong programming skills in Python and experience building production software services.
  • Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.
  • Strong understanding of machine learning evaluation methodologies and performance measurement.
  • Experience developing reproducible experiments and benchmark datasets.
  • Familiarity with large language models and modern LLM application architectures.
  • Understanding of prompt engineering, model inference, and AI application integration patterns.
  • Experience evaluating model outputs for quality, accuracy, and reliability.
  • Familiarity with retrieval‑augmented generation and retrieval evaluation methods.
  • Experience working with structured datasets, data pipelines, and analytical workflows.
  • Strong understanding of software testing, regression testing, and continuous integration.
  • Experience building or maintaining backend services and APIs.
  • Familiarity with cloud infrastructure and production deployment environments.
  • Experience with monitoring, logging, tracing, and production troubleshooting.
  • Understanding of model latency, throughput, and operational performance metrics.
  • Ability to investigate complex system failures and identify root causes.
  • Experience designing metrics and reporting systems for technical performance.
  • Familiarity with version control, code review, and collaborative software development.
  • Understanding of data quality and its impact on model evaluation.
  • Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost.
  • Strong analytical skills and attention to detail.
  • Experience collaborating with engineers, product managers, and technical stakeholders.
  • Ability to independently own complex engineering initiatives.
  • Strong communication skills and the ability to explain evaluation results clearly.

Responsibilities

  • Design and build evaluation infrastructure for AI and machine learning applications across Bask Health.
  • Develop automated testing systems for large language model applications and intelligent workflows.
  • Establish measurable quality standards for production AI features.
  • Create evaluation pipelines that assess accuracy, relevance, completeness, consistency, and reliability.
  • Build and maintain representative datasets for testing AI application behavior.
  • Develop regression testing systems to detect quality changes when models, prompts, or application logic are updated.
  • Design evaluation methods for retrieval‑augmented generation and knowledge‑grounded AI systems.
  • Measure retrieval quality, grounding accuracy, and unsupported model outputs.
  • Build tools for identifying hallucinations, incorrect reasoning, and unreliable responses.
  • Develop testing approaches for AI agents, tool calling, and multi‑step workflows.
  • Evaluate whether AI systems follow application instructions and respect workflow boundaries.
  • Implement automated checks for structured outputs, API interactions, and tool execution.
  • Build monitoring systems that track AI performance in production.
  • Develop dashboards and reporting tools for model quality, latency, errors, and operational reliability.
  • Investigate production AI failures and identify their underlying technical causes.
  • Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations.
  • Establish testing procedures for model upgrades and provider changes.
  • Develop benchmarking tools to compare models based on quality, performance, and cost.
  • Implement evaluation processes for sensitive healthcare‑related AI workflows.
  • Work with product and clinical stakeholders to define appropriate acceptance criteria and escalation requirements.
  • Support human review and feedback collection for AI outputs that require additional validation.
  • Design reproducible experiments for evaluating AI system improvements.
  • Build reusable evaluation libraries, services, and internal engineering tools.
  • Integrate AI evaluation checks into development and deployment workflows.
  • Improve observability through structured logging, tracing, and production diagnostics.
  • Mentor engineers on AI testing methodologies and reliability practices.
  • Contribute to architecture discussions involving AI infrastructure, testing, and monitoring.
  • Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance.

Skills

Python
LLM evaluation
AI reliability
Production ML systems

Tools

LangSmith
DeepEval
Ragas
AWS
Cursor
Claude Code

Job description

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design and maintain systems that measure and improve AI quality across healthcare workflows.

You'll build evaluation pipelines, regression tests, and monitoring to ensure safe, reliable AI features. This remote US role offers ownership of critical reliability practices and collaboration with product and clinical stakeholders.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior ML Reliability Engineer - AI Evaluation & HealthTech
Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health • New York (NY)

On-site
USD 180,000 - 240,000
Senior ML Engineer, AI Evaluation & Reliability
Senior ML Engineer, AI Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000
AI Evaluation & Reliability Engineer
AI Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 140,000 - 200,000
AI Reliability Engineer — Evaluation & Testing
AI Reliability Engineer — Evaluation & Testing

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
Senior Healthtech AI Engineer — Production LLMs (Remote)
Senior Healthtech AI Engineer — Production LLMs (Remote)

Bask Health • United States

Remote
USD 160,000 - 220,000
AI Evaluation & Quality Assurance Engineer
AI Evaluation & Quality Assurance Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
AI Reliability & Evaluation Engineer — Remote
AI Reliability & Evaluation Engineer — Remote

Techventus • United States

Remote
USD 8,900 - 18,000
Remote-first working arrangements
Opportunity to specialize in AIReliabl
Exposure to production AI systems
+1
Senior ML Evaluation Engineer – LLM & AI Reliability
Senior ML Evaluation Engineer – LLM & AI Reliability

Bask Health • United States

Remote
USD 160,000 - 220,000
Remote Senior Software Engineer, LLM Evaluation & QA Tools
Remote Senior Software Engineer, LLM Evaluation & QA Tools

OpenTrain AI, Inc. • United States

Remote
USD 83,000 - 124,000
Senior Machine Learning Engineer, Evaluation & Reliability
Senior Machine Learning Engineer, Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000