AI Evaluation & Reliability Engineer

Bask Health

United States

Remote

USD 140,000 - 200,000

Full time

6 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Bask Health is seeking a Machine Learning Engineer, Evaluation & Reliability to build evaluation infrastructure and testing workflows for AI-powered features across the platform. You will measure model performance, identify failures, and verify improvements in real-world usage while ensuring privacy and safety constraints in healthcare contexts.

You’ll collaborate with AI engineers, backend engineers, and product teams to translate expectations into measurable tests and robust engineering

Qualifications

  • 3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field.
  • Experience building and maintaining production software or machine learning applications.
  • Strong programming skills in Python, TypeScript, or comparable languages.
  • Familiarity with large language models and modern generative AI applications.
  • Experience testing or evaluating machine learning models or AI-powered software.
  • Understanding of machine learning evaluation metrics and experimental methodologies.
  • Experience developing automated tests and validation pipelines.
  • Familiarity with prompt engineering and model output analysis.
  • Ability to analyze structured and unstructured datasets.
  • Experience writing scripts or services for data processing and evaluation.
  • Understanding of model accuracy, precision, recall, and other performance measures where applicable.
  • Familiarity with API integrations and backend application development.
  • Experience working with relational databases or analytical data systems.
  • Ability to identify patterns in model failures and translate findings into engineering improvements.
  • Understanding of regression testing and software quality assurance.
  • Familiarity with logging, monitoring, and production troubleshooting.
  • Experience using Git and collaborative software development workflows.
  • Understanding of the limitations and failure modes of large language models.
  • Ability to design representative test cases and evaluation scenarios.
  • Familiarity with cloud-based development and deployment environments.
  • Experience writing maintainable, well-tested production code.
  • Strong analytical thinking and attention to detail.
  • Ability to communicate technical evaluation results clearly.
  • Experience collaborating with engineers, product managers, and cross-functional teams.
  • Nice to Have: Experience building evaluation frameworks for large language models.
  • Familiarity with OpenAI, Anthropic, or comparable model APIs.
  • Experience with tools such as LangSmith, DeepEval, Ragas, or comparable AI evaluation frameworks.
  • Familiarity with LLM-as-a-judge evaluation techniques.
  • Experience creating human-labeled evaluation datasets.
  • Understanding of inter-rater agreement and evaluation consistency.
  • Experience evaluating retrieval-augmented generation systems.
  • Familiarity with embeddings, semantic search, and vector databases.
  • Experience evaluating AI agents and tool-calling workflows.
  • Familiarity with prompt injection testing and AI security evaluations.
  • Experience implementing automated regression testing for AI applications.
  • Familiarity with experiment tracking and model comparison tools.
  • Experience building dashboards for machine learning performance monitoring.
  • Understanding of statistical analysis and experiment design.
  • Experience working with Python data analysis libraries.
  • Familiarity with TypeScript and Node.js application development.
  • Experience working with AWS or serverless infrastructure.
  • Experience evaluating AI systems used in healthcare or other sensitive environments.
  • Understanding of HIPAA and healthcare data privacy considerations.
  • Experience using AI-assisted development tools such as Cursor or Claude Code.

Responsibilities

  • Build and maintain evaluation systems for AI-powered applications across Bask Health.
  • Develop automated testing frameworks for large language models and AI application workflows.
  • Create and maintain representative datasets for evaluating model behavior.
  • Implement benchmarks that measure response accuracy, relevance, completeness, and consistency.
  • Develop evaluation pipelines for prompts, retrieval systems, and model configurations.
  • Test AI assistants and agents against realistic user scenarios and workflow requirements.
  • Build automated regression tests to detect performance changes between releases.
  • Measure hallucination rates, unsupported claims, and other model quality failures.
  • Evaluate retrieval-augmented generation systems for document relevance and answer grounding.
  • Develop tests for structured model outputs and schema compliance.
  • Evaluate tool-calling accuracy and agent action selection.
  • Test AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.
  • Implement evaluation checks for sensitive patient and operational information.
  • Build dashboards and reports that communicate AI quality metrics to engineering teams.
  • Analyze evaluation results and identify recurring failure patterns.
  • Investigate production incidents involving inaccurate or unreliable AI outputs.
  • Collaborate with applied AI engineers to improve prompts, retrieval logic, and application behavior.
  • Support model comparisons and controlled experiments.
  • Evaluate tradeoffs between model quality, latency, reliability, and inference cost.
  • Develop automated quality checks for CI/CD and AI feature release workflows.
  • Implement monitoring and alerting for production AI performance.
  • Build tools for reviewing, labeling, and analyzing model responses.
  • Help maintain evaluation datasets as products and user behavior evolve.
  • Document evaluation methodologies, test cases, and known model limitations.
  • Work with product teams to define measurable acceptance criteria for AI features.
  • Collaborate with senior engineers on evaluation standards and reliability improvements.
  • Participate in code reviews, technical discussions, and engineering planning.
  • Own assigned evaluation and reliability projects from implementation through deployment and ongoing improvement.
  • 3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field.

Skills

Python
TypeScript
Machine learning
Evaluation
Data analysis
APIs
Git
Cloud
AWS
Testing
LLMs
Prompt engineering
Retrieval systems
Back-end development

Tools

Cursor
Claude Code
LangSmith
DeepEval
Ragas

Job description

Bask Health is seeking a Machine Learning Engineer, Evaluation & Reliability to build evaluation infrastructure and testing workflows for AI-powered features across the platform. You will measure model performance, identify failures, and verify improvements in real-world usage while ensuring privacy and safety constraints in healthcare contexts.

You’ll collaborate with AI engineers, backend engineers, and product teams to translate expectations into measurable tests and robust engineering

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Reliability Engineer — Evaluation & Testing
AI Reliability Engineer — Evaluation & Testing

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
Senior ML Reliability Engineer - AI Evaluation & HealthTech
Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health • New York (NY)

On-site
USD 180,000 - 240,000
Senior ML Engineer, AI Evaluation & Reliability
Senior ML Engineer, AI Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000
AI Evaluation & Quality Assurance Engineer
AI Evaluation & Quality Assurance Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
AI Evaluation Platform Engineer
AI Evaluation Platform Engineer

Bask Health • New York (NY)

On-site
USD 140,000 - 190,000
Remote Senior LLM Evaluation & Reliability Engineer
Remote Senior LLM Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 160,000 - 220,000
Healthcare AI Quality Assurance Engineer
Healthcare AI Quality Assurance Engineer

Bask Health • United States

Remote
USD 120,000 - 180,000
AI Quality & Evaluation Engineer — Backend Tools
AI Quality & Evaluation Engineer — Backend Tools

Bask Health • United States

Remote
USD 120,000 - 180,000
AI Quality & Evaluation Engineer
AI Quality & Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
Machine Learning Engineer, Evaluation & Reliability
Machine Learning Engineer, Evaluation & Reliability

Bask Health • United States

Remote
USD 140,000 - 200,000