AI Evaluation Platform Engineer

Ellipsis Health Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 160,000 - 210,000; Full time

Benefits offered by this job

401(k) matching

Health insurance

Flex time off

Job summary

Ellipsis Health Inc. in San Francisco is seeking a Software Engineer to join the AI Evaluations team.

You will build the infrastructure and tooling that enable reliable, repeatable evaluation of AI systems in production, including APIs, data integrations, and reporting tools for scalable evaluations of ASR, LLMs, and TTS components. You will also create debugging and observability tools, work with ML, engineering and QA teams, and ensure maintainable, developer-friendly systems that support

Qualifications

• 5+ years of professional software engineering experience building backend systems, platforms, or developer tooling.
• Strong production-grade infrastructure experience with APIs, services, and data pipelines.
• Experience with test automation, evaluation pipelines, or CI/CD-integrated testing.

Responsibilities

• Build and maintain infrastructure and tooling for the AI evaluations platform used by internal teams.
• Develop and productionalize evaluation frameworks for ASR, LLMs, TTS, knowledge bases, and guardrails.
• Collaborate with ML, engineering and QA to translate evaluation requirements into robust infrastructure.
• Improve developer experience by making evaluation systems easy to extend and well-documented.

Skills

Backend systems

DevOps tooling

Observability

Python

CI/CD

Tools

Langfuse

CI/CD pipelines

Job description

Ellipsis Health Inc. in San Francisco is seeking a Software Engineer to join the AI Evaluations team.

You will build the infrastructure and tooling that enable reliable, repeatable evaluation of AI systems in production, including APIs, data integrations, and reporting tools for scalable evaluations of ASR, LLMs, and TTS components. You will also create debugging and observability tools, work with ML, engineering and QA teams, and ensure maintainable, developer-friendly systems that support