Senior ML Evaluation Engineer – LLM & AI Reliability

Bask Health

United States

Remote

USD 160,000 - 220,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Bask Health is seeking a Senior Machine Learning Engineer, Evaluation & Reliability to design and maintain systems that measure and improve AI quality across healthcare workflows.

You'll build evaluation pipelines, regression tests, and monitoring to ensure safe, reliable AI features. This remote US role offers ownership of critical reliability practices and collaboration with product and clinical stakeholders.

Qualifications

  • 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline.
  • Demonstrated experience developing and operating machine learning or AI systems in production.
  • Strong programming skills in Python and experience building production software services.
  • Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.
  • Strong understanding of machine learning evaluation methodologies and performance measurement.
  • Experience developing reproducible experiments and benchmark datasets.
  • Familiarity with large language models and modern LLM application architectures.
  • Understanding of prompt engineering, model inference, and AI application integration patterns.
  • Experience evaluating model outputs for quality, accuracy, and reliability.
  • Familiarity with retrieval‑augmented generation and retrieval evaluation methods.
  • Experience working with structured datasets, data pipelines, and analytical workflows.
  • Strong understanding of software testing, regression testing, and continuous integration.
  • Experience building or maintaining backend services and APIs.
  • Familiarity with cloud infrastructure and production deployment environments.
  • Experience with monitoring, logging, tracing, and production troubleshooting.
  • Understanding of model latency, throughput, and operational performance metrics.
  • Ability to investigate complex system failures and identify root causes.
  • Experience designing metrics and reporting systems for technical performance.
  • Familiarity with version control, code review, and collaborative software development.
  • Understanding of data quality and its impact on model evaluation.
  • Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost.
  • Strong analytical skills and attention to detail.
  • Experience collaborating with engineers, product managers, and technical stakeholders.
  • Ability to independently own complex engineering initiatives.
  • Strong communication skills and the ability to explain evaluation results clearly.

Responsibilities

  • Design and build evaluation infrastructure for AI and machine learning applications across Bask Health.
  • Develop automated testing systems for large language model applications and intelligent workflows.
  • Establish measurable quality standards for production AI features.
  • Create evaluation pipelines that assess accuracy, relevance, completeness, consistency, and reliability.
  • Build and maintain representative datasets for testing AI application behavior.
  • Develop regression testing systems to detect quality changes when models, prompts, or application logic are updated.
  • Design evaluation methods for retrieval‑augmented generation and knowledge‑grounded AI systems.
  • Measure retrieval quality, grounding accuracy, and unsupported model outputs.
  • Build tools for identifying hallucinations, incorrect reasoning, and unreliable responses.
  • Develop testing approaches for AI agents, tool calling, and multi‑step workflows.
  • Evaluate whether AI systems follow application instructions and respect workflow boundaries.
  • Implement automated checks for structured outputs, API interactions, and tool execution.
  • Build monitoring systems that track AI performance in production.
  • Develop dashboards and reporting tools for model quality, latency, errors, and operational reliability.
  • Investigate production AI failures and identify their underlying technical causes.
  • Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations.
  • Establish testing procedures for model upgrades and provider changes.
  • Develop benchmarking tools to compare models based on quality, performance, and cost.
  • Implement evaluation processes for sensitive healthcare‑related AI workflows.
  • Work with product and clinical stakeholders to define appropriate acceptance criteria and escalation requirements.
  • Support human review and feedback collection for AI outputs that require additional validation.
  • Design reproducible experiments for evaluating AI system improvements.
  • Build reusable evaluation libraries, services, and internal engineering tools.
  • Integrate AI evaluation checks into development and deployment workflows.
  • Improve observability through structured logging, tracing, and production diagnostics.
  • Mentor engineers on AI testing methodologies and reliability practices.
  • Contribute to architecture discussions involving AI infrastructure, testing, and monitoring.
  • Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance.

Skills

Python
LLM evaluation
AI reliability
Production ML systems

Tools

LangSmith
DeepEval
Ragas
AWS
Cursor
Claude Code

Job description

Senior ML Evaluation Engineer – LLM & AI Reliability — Remote

This is a high-impact, emerging-niche role at the intersection of LLM engineering and production reliability — a skill set in explosive demand as companies race to ship AI features responsibly. Senior ML Evaluation Engineers are relatively rare, and this specialization positions you at the frontier of AI quality assurance.

Compensation benchmark: No salary is listed, but Senior ML Engineers in the US typically earn $160,000–$220,000+ in total comp at well-funded startups. The absence of disclosed pay warrants a direct conversation about equity and total compensation early in the process.

  • LLM evaluation frameworks (LangSmith, DeepEval, Ragas) — transferable across the entire AI industry
  • RAG system quality assessment — a highly sought-after specialization in 2024–2025
  • AI observability and production monitoring — foundational for any ML platform role
  • Healthcare AI compliance (HIPAA-adjacent) — a differentiator in a regulated, high-growth vertical

Culture signals: Bask describes itself as "entrepreneurial" and "builder-led" — classic early-stage startup language. Engineers own features end-to-end and interact with leadership directly, suggesting a small team with high individual leverage and visibility.

Green flags: Genuine ownership, AI-first tooling culture (Cursor, Claude Code), and a niche specialization that will only grow in demand. Red flags to watch: No salary transparency, no mention of benefits or PTO, and the breadth of the responsibilities list suggests a team that may be understaffed in this area.

About Bask Health

Bask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses.

Our platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place.

We're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform.

Engineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality.

About the Role

We're looking for a Senior Machine Learning Engineer, Evaluation & Reliability to design, build, and maintain the systems that measure and improve the quality of Bask Health's AI-powered applications.

As Bask introduces more AI capabilities across patient experiences, merchant tools, clinical workflows, and internal operations, we need reliable ways to understand how these systems perform. You'll help establish the engineering infrastructure, evaluation methods, and monitoring practices that allow teams to ship AI features with confidence.

This role sits at the intersection of machine learning engineering, software infrastructure, automated testing, and production reliability. You'll work on evaluation pipelines, benchmark datasets, regression testing, model monitoring, and tools that help engineers identify failures before they affect users.

You'll develop systems that measure more than whether an AI model produces a response — you'll help determine whether that response is accurate, relevant, grounded in appropriate information, consistent with application requirements, and safe for its intended use.

You'll also investigate production issues, identify patterns in model failures, and collaborate with applied AI engineers to improve prompts, retrieval systems, model configurations, and application behavior.

Because Bask operates in healthcare-related workflows, reliability is especially important. You'll help establish safeguards for sensitive use cases, support human review where appropriate, and make sure AI systems are evaluated against clear quality expectations before deployment.

This is a hands‑on senior engineering position with ownership over evaluation infrastructure and reliability practices. You'll contribute production code, design scalable testing systems, and help establish standards that other engineers can use across multiple AI applications.

We're looking for someone who enjoys turning complex machine learning behavior into measurable engineering systems and who understands that successful AI products depend on consistent performance, not just impressive demonstrations.

What You'll Do
  • Design and build evaluation infrastructure for AI and machine learning applications across Bask Health.
  • Develop automated testing systems for large language model applications and intelligent workflows.
  • Establish measurable quality standards for production AI features.
  • Create evaluation pipelines that assess accuracy, relevance, completeness, consistency, and reliability.
  • Build and maintain representative datasets for testing AI application behavior.
  • Develop regression testing systems to detect quality changes when models, prompts, or application logic are updated.
  • Design evaluation methods for retrieval‑augmented generation and knowledge‑grounded AI systems.
  • Measure retrieval quality, grounding accuracy, and unsupported model outputs.
  • Build tools for identifying hallucinations, incorrect reasoning, and unreliable responses.
  • Develop testing approaches for AI agents, tool calling, and multi‑step workflows.
  • Evaluate whether AI systems follow application instructions and respect workflow boundaries.
  • Implement automated checks for structured outputs, API interactions, and tool execution.
  • Build monitoring systems that track AI performance in production.
  • Develop dashboards and reporting tools for model quality, latency, errors, and operational reliability.
  • Investigate production AI failures and identify their underlying technical causes.
  • Collaborate with applied AI engineers to improve prompts, retrieval pipelines, and model configurations.
  • Establish testing procedures for model upgrades and provider changes.
  • Develop benchmarking tools to compare models based on quality, performance, and cost.
  • Implement evaluation processes for sensitive healthcare‑related AI workflows.
  • Work with product and clinical stakeholders to define appropriate acceptance criteria and escalation requirements.
  • Support human review and feedback collection for AI outputs that require additional validation.
  • Design reproducible experiments for evaluating AI system improvements.
  • Build reusable evaluation libraries, services, and internal engineering tools.
  • Integrate AI evaluation checks into development and deployment workflows.
  • Improve observability through structured logging, tracing, and production diagnostics.
  • Mentor engineers on AI testing methodologies and reliability practices.
  • Contribute to architecture discussions involving AI infrastructure, testing, and monitoring.
  • Own evaluation and reliability improvements from technical design through deployment and ongoing maintenance.
Requirements
  • 5+ years of professional experience in machine learning engineering, software engineering, or a closely related technical discipline.
  • Demonstrated experience developing and operating machine learning or AI systems in production.
  • Strong programming skills in Python and experience building production software services.
  • Experience designing automated testing frameworks or evaluation pipelines for AI or machine learning applications.
  • Strong understanding of machine learning evaluation methodologies and performance measurement.
  • Experience developing reproducible experiments and benchmark datasets.
  • Familiarity with large language models and modern LLM application architectures.
  • Understanding of prompt engineering, model inference, and AI application integration patterns.
  • Experience evaluating model outputs for quality, accuracy, and reliability.
  • Familiarity with retrieval‑augmented generation and retrieval evaluation methods.
  • Experience working with structured datasets, data pipelines, and analytical workflows.
  • Strong understanding of software testing, regression testing, and continuous integration.
  • Experience building or maintaining backend services and APIs.
  • Familiarity with cloud infrastructure and production deployment environments.
  • Experience with monitoring, logging, tracing, and production troubleshooting.
  • Understanding of model latency, throughput, and operational performance metrics.
  • Ability to investigate complex system failures and identify root causes.
  • Experience designing metrics and reporting systems for technical performance.
  • Familiarity with version control, code review, and collaborative software development.
  • Understanding of data quality and its impact on model evaluation.
  • Ability to evaluate technical tradeoffs between quality, performance, reliability, and cost.
  • Strong analytical skills and attention to detail.
  • Experience collaborating with engineers, product managers, and technical stakeholders.
  • Ability to independently own complex engineering initiatives.
  • Strong communication skills and the ability to explain evaluation results clearly.
Nice to Have
  • Experience building evaluation frameworks specifically for LLM-powered applications.
  • Familiarity with tools such as LangSmith, DeepEval, Ragas, or comparable evaluation platforms.
  • Experience implementing LLM-as-a-judge evaluation methods and validating their limitations.
  • Experience building automated hallucination detection or groundedness evaluation systems.
  • Familiarity with agent evaluation and multi‑step workflow testing.
  • Experience evaluating tool‑calling accuracy and structured model outputs.
  • Experience with retrieval metrics such as precision, recall, and ranking quality.
  • Familiarity with vector databases, embeddings, and semantic search.
  • Experience with MLflow, Weights & Biases, or similar experiment tracking tools.
  • Experience developing machine learning observability infrastructure.
  • Familiarity with model drift detection and data quality monitoring.
  • Experience integrating evaluation checks into CI/CD pipelines.
  • Experience with distributed processing or large‑scale evaluation workloads.
  • Familiarity with AWS and cloud‑based machine learning infrastructure.
  • Experience building internal developer tools for AI engineering teams.
  • Experience evaluating systems built with OpenAI, Anthropic, or comparable model providers.
  • Familiarity with healthcare, telehealth, or other sensitive‑data applications.
  • Understanding of HIPAA and healthcare data privacy considerations.
  • Experience designing human‑in‑the‑loop evaluation or review systems.
  • Experience using AI‑assisted development tools such as Cursor or Claude Code.

Is this role fully remote or does it require on‑site presence? The role is listed as remote within the United States. There is no mention of office requirements or mandatory on‑site days in the job description. What experience level is required for this position? Bask Health requires 5+ years of professional experience in machine learning engineering, software engineering, or a closely related discipline, along with demonstrated production ML/AI system experience. Is salary information available for this role? No salary range is disclosed in the job posting. Candidates should raise compensation expectations directly with the hiring team during the interview process. What AI tools and frameworks will I work with in this role? You'll work with LLM evaluation tools (LangSmith, DeepEval, Ragas), AI‑assisted development tools like Cursor and Claude Code, and cloud infrastructure (preferably AWS). Experience with OpenAI and Anthropic model providers is a plus. Does this role require healthcare or HIPAA experience? Healthcare experience is listed as "nice to have" rather than required, but familiarity with HIPAA and sensitive‑data applications is beneficial given Bask Health's telehealth context.

Remote work & career advice that actually works.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Applied AI Engineer - Healthtech LLM Role
Senior Applied AI Engineer - Healthtech LLM Role

Bask Health • United States

Remote
USD 160,000 - 220,000
Senior Machine Learning Engineer, Evaluation & Reliability
Senior Machine Learning Engineer, Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000
Machine Learning Engineer, Evaluation & Reliability
Machine Learning Engineer, Evaluation & Reliability

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
Senior Machine Learning Engineer, Evaluation & Reliability
Senior Machine Learning Engineer, Evaluation & Reliability

Bask Health • New York (NY)

On-site
USD 180,000 - 240,000
Machine Learning Engineer, Evaluation & Reliability
Machine Learning Engineer, Evaluation & Reliability

Bask Health • United States

Remote
USD 140,000 - 200,000
Senior Platform Engineer - Healthtech API & AWS Role
Senior Platform Engineer - Healthtech API & AWS Role

Bask Health • United States

Remote
USD 150,000 - 200,000
Senior Backend Engineer, EMR & Care Platform | Bask
Senior Backend Engineer, EMR & Care Platform | Bask

Bask Health • United States

Remote
USD 160,000 - 230,000
Senior Applied AI Engineer, Patient Experience
Senior Applied AI Engineer, Patient Experience

Bask Health • United States

On-site
USD 140,000 - 210,000
Senior Full Stack Engineer - Healthtech SaaS Career Growth
Senior Full Stack Engineer - Healthtech SaaS Career Growth

Bask Health • United States

Remote
USD 145,000 - 195,000
Software Engineer, AI Quality & Evaluation
Software Engineer, AI Quality & Evaluation

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000