AI Quality & Evaluation Engineer — Backend Tools

Bask Health

United States

Remote

USD 120,000 - 180,000

Full time

6 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Bask Health is seeking a Software Engineer, AI Quality & Evaluation to design and build internal tools and backend services for testing AI-powered applications. You will implement automated evaluation pipelines, dataset management, and interfaces for engineers to inspect AI behaviors, with a focus on reliability and privacy in healthcare contexts.

You’ll collaborate with AI/ML engineers and backend teams to integrate testing into development workflows, while improving infrastructure performance

Qualifications

  • 3+ years of professional software engineering experience, backend/platform/infrastructure roles.
  • Strong programming skills in TypeScript, JavaScript, or Python.
  • Experience designing and developing APIs and service integrations.
  • Familiarity with automated testing frameworks and software quality practices.
  • Understanding of CI/CD workflows and production deployments.
  • Experience with large language models and AI testing is a plus.
  • Familiarity with relational databases and data-backed applications.
  • Knowledge of multi-tenant SaaS and data privacy considerations.

Responsibilities

  • Design, build, and maintain software infrastructure for AI quality evaluation.
  • Develop backend services to execute/ manage automated AI evaluations.
  • Create reusable testing tools for LLM applications.
  • Build APIs for submitting evaluation jobs and retrieving results.
  • Develop systems to manage evaluation datasets, test cases, and outcomes.
  • Implement regression testing and CI/CD integrated evaluation checks.
  • Develop dashboards to review evaluation outcomes and results over time.
  • Collaborate with AI/ML engineers to translate testing needs into tooling.
  • Ensure data privacy, access controls, and tenant isolation in evaluation infra.
  • Investigate failures and optimize performance and reliability.

Skills

TypeScript/JavaScript
Python
APIs
Automated testing
CI/CD
LLM testing
Cloud services
Distributed systems

Tools

LangSmith
DeepEval
Ragas
promptfoo

Job description

Bask Health is seeking a Software Engineer, AI Quality & Evaluation to design and build internal tools and backend services for testing AI-powered applications. You will implement automated evaluation pipelines, dataset management, and interfaces for engineers to inspect AI behaviors, with a focus on reliability and privacy in healthcare contexts.

You’ll collaborate with AI/ML engineers and backend teams to integrate testing into development workflows, while improving infrastructure performance

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Quality & Evaluation Engineer
AI Quality & Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
AI Evaluation & Quality Assurance Engineer
AI Evaluation & Quality Assurance Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
Healthcare AI Quality Assurance Engineer
Healthcare AI Quality Assurance Engineer

Bask Health • United States

Remote
USD 120,000 - 180,000
AI Evaluation & Reliability Engineer
AI Evaluation & Reliability Engineer

Bask Health • United States

Remote
USD 140,000 - 200,000
Senior ML Engineer, AI Evaluation & Reliability
Senior ML Engineer, AI Evaluation & Reliability

Bask Health • United States

Remote
USD 150,000 - 210,000
AI Reliability Engineer — Evaluation & Testing
AI Reliability Engineer — Evaluation & Testing

Bask Health • New York (NY)

On-site
USD 120,000 - 190,000
AI Platform Engineer — Backend Infra & AI Services
AI Platform Engineer — Backend Infra & AI Services

Bask Health • United States

Remote
USD 120,000 - 170,000
Senior ML Reliability Engineer - AI Evaluation & HealthTech
Senior ML Reliability Engineer - AI Evaluation & HealthTech

Bask Health • New York (NY)

On-site
USD 180,000 - 240,000
AI Evaluation Platform Engineer
AI Evaluation Platform Engineer

Bask Health • New York (NY)

On-site
USD 140,000 - 190,000
AI Evaluation Engineer
AI Evaluation Engineer

Bask Health • United States

Remote
USD 120,000 - 180,000