AI Reliability Engineer — Evaluation & Testing

Bask Health; New York (NY); On-site; USD 120,000 - 190,000; Full time

Job summary

Bask Health is seeking a Machine Learning Engineer, Evaluation & Reliability to ensure AI-powered features are accurate, dependable, and ready for real-world use. You will help develop evaluation infrastructure, testing workflows, and performance metrics for large language models, retrieval systems, AI assistants, and automated workflows.

This hands-on role combines machine learning evaluation, software development, data analysis, and production QA.

Qualifications

• 3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a closely related technical field.
• Experience building and maintaining production software or machine learning applications.
• Strong programming skills in Python, TypeScript, or comparable languages.
• Familiarity with large language models and modern generative AI applications.
• Experience testing or evaluating machine learning models or AI-powered software.
• Understanding of machine learning evaluation metrics and experimental methodologies.
• Experience developing automated tests and validation pipelines.
• Familiarity with prompt engineering and model output analysis.
• Ability to analyze structured and unstructured datasets.
• Experience writing scripts or services for data processing and evaluation.
• Understanding of model accuracy, precision, recall, and other performance measures where applicable.
• Familiarity with API integrations and backend application development.
• Experience working with relational databases or analytical data systems.
• Ability to identify patterns in model failures and translate findings into engineering improvements.
• Understanding of regression testing and software quality assurance.
• Familiarity with logging, monitoring, and production troubleshooting.
• Experience using Git and collaborative software development workflows.
• Understanding of the limitations and failure modes of large language models.
• Ability to design representative test cases and evaluation scenarios.
• Familiarity with cloud-based development and deployment environments.
• Experience writing maintainable, well‑tested production code.
• Strong analytical thinking and attention to detail.
• Ability to communicate technical evaluation results clearly.
• Experience collaborating with engineers, product managers, and cross-functional teams.
• Ability to independently own assigned technical projects.

Responsibilities

• Build and maintain evaluation systems for AI-powered applications across Bask Health.
• Develop automated testing frameworks for large language models and AI application workflows.
• Create and maintain representative datasets for evaluating model behavior.
• Implement benchmarks that measure response accuracy, relevance, completeness, and consistency.
• Develop evaluation pipelines for prompts, retrieval systems, and model configurations.
• Test AI assistants and agents against realistic user scenarios and workflow requirements.
• Build automated regression tests to detect performance changes between releases.
• Measure hallucination rates, unsupported claims, and other model quality failures.
• Evaluate retrieval-augmented generation systems for document relevance and answer grounding.
• Develop tests for structured model outputs and schema compliance.
• Evaluate tool‑calling accuracy and agent action selection.
• Test AI workflows for unauthorized actions, incorrect permissions, and unsafe behavior.
• Implement evaluation checks for sensitive patient and operational information.
• Build dashboards and reports that communicate AI quality metrics to engineering teams.
• Analyze evaluation results and identify recurring failure patterns.
• Investigate production incidents involving inaccurate or unreliable AI outputs.
• Collaborate with applied AI engineers to improve prompts, retrieval logic, and application behavior.
• Support model comparisons and controlled experiments.
• Evaluate tradeoffs between model quality, latency, reliability, and inference cost.
• Develop automated quality checks for CI/CD and AI feature release workflows.
• Implement monitoring and alerting for production AI performance.
• Build tools for reviewing, labeling, and analyzing model responses.
• Help maintain evaluation datasets as products and user behavior evolve.
• Document evaluation methodologies, test cases, and known model limitations.
• Work with product teams to define measurable acceptance criteria for AI features.
• Collaborate with senior engineers on evaluation standards and reliability improvements.
• Participate in code reviews, technical discussions, and engineering planning.
• Own assigned evaluation and reliability projects from implementation through deployment and ongoing improvement.

Skills

Python

TypeScript

ML evaluation

Testing frameworks

Data analysis

CI/CD

Telemetry

Model failures

Experiment design

Cloud development

Git

Communication

Tools

Cursor

Claude Code

Job description

Bask Health is seeking a Machine Learning Engineer, Evaluation & Reliability to ensure AI-powered features are accurate, dependable, and ready for real-world use. You will help develop evaluation infrastructure, testing workflows, and performance metrics for large language models, retrieval systems, AI assistants, and automated workflows.

This hands-on role combines machine learning evaluation, software development, data analysis, and production QA.